Using a download cache
======================

Normally, when distributions are installed, if any processing is
needed, they are downloaded from the internet to a temporary directory
and then installed from there.  A download cache can be used to avoid
the download step.  This can be useful to reduce network access and to
create source distributions of an entire buildout.

The buildout download-cache option can be used to specify a directory
to be used as a download cache.

In this example, we'll create a directory to hold the cache:

    >>> cache = tmpdir('cache')

And set up a buildout that downloads some eggs:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = eggs
    ... download-cache = %(cache)s
    ... find-links = %(link_server)s
    ...
    ... [eggs]
    ... recipe = zc.recipe.egg
    ... eggs = demo ==0.2
    ... ''' % globals())

We specified a link server that has some distributions available for
download:

    >>> print_(get(link_server), end='')
    <html><body>
    <a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br>
    <a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br>
    <a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br>
    <a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br>
    <a href="demo-0.4rc1-py2.4.egg">demo-0.4rc1-py2.4.egg</a><br>
    <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>
    <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>
    <a href="demoneeded-1.2rc1.zip">demoneeded-1.2rc1.zip</a><br>
    <a href="du_zipped-1.0-pyN.N.egg">du_zipped-1.0-pyN.N.egg</a><br>
    <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>
    <a href="index/">index/</a><br>
    <a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br>
    </body></html>


We'll enable logging on the link server so we can see what's going on:

    >>> _ = get(link_server+'enable_server_logging')
    GET 200 /enable_server_logging

We also specified a download cache.

If we run the buildout, we'll see the eggs installed from the link
server as usual:

    >>> print_(system(buildout), end='')
    GET 200 /
    GET 200 /demo-0.2-py2.4.egg
    GET 200 /demoneeded-1.1.zip
    Installing eggs.
    Getting distribution for 'demo==0.2'.
    Got demo 0.2.
    Getting distribution for 'demoneeded'.
    Got demoneeded 1.1.
    Generated script '/sample-buildout/bin/demo'.

We'll also get the download cache populated.  The buildout doesn't put
files in the cache directly.  It creates an intermediate directory,
dist:


    >>> ls(cache)
    d  dist

    >>> ls(cache, 'dist')
    -  demo-0.2-py2.4.egg
    -  demoneeded-1.1.zip

If we remove the installed eggs from eggs directory and re-run the buildout:

    >>> import os
    >>> for  f in os.listdir('eggs'):
    ...     if f.startswith('demo'):
    ...         remove('eggs', f)

    >>> print_(system(buildout), end='')
    GET 200 /
    Updating eggs.
    Getting distribution for 'demo==0.2'.
    Got demo 0.2.
    Getting distribution for 'demoneeded'.
    Got demoneeded 1.1.

We see that the distributions aren't downloaded, because they're
downloaded from the cache.

Installing solely from a download cache
---------------------------------------

A download cache can be used as the basis of application source
releases.  In an application source release, we want to distribute an
application that can be built without making any network accesses.  In
this case, we distribute a buildout with download cache and tell the
buildout to install from the download cache only, without making
network accesses.  The buildout install-from-cache option can be used
to signal that packages should be installed only from the download
cache.

Let's remove our installed eggs and run the buildout with the
install-from-cache option set to true:

    >>> for  f in os.listdir('eggs'):
    ...     if f.startswith('demo'):
    ...         remove('eggs', f)

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = eggs
    ... download-cache = %(cache)s
    ... install-from-cache = true
    ... find-links = %(link_server)s
    ...
    ... [eggs]
    ... recipe = zc.recipe.egg
    ... eggs = demo
    ... ''' % globals())

    >>> print_(system(buildout), end='')
    Uninstalling eggs.
    Installing eggs.
    Getting distribution for 'demo'.
    Got demo 0.2.
    Getting distribution for 'demoneeded'.
    Got demoneeded 1.1.
    Generated script '/sample-buildout/bin/demo'.

Auto-creation of download cache directory
-----------------------------------------

With zc.buildout version 2.2.2 or higher the cache directory is automatically
created::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts =
    ... download-cache = %(cache)s/newdir
    ... ''' % globals())

    >>> print_(system(buildout), end='')
    Creating directory '/cache/newdir'.
    Uninstalling eggs.

    >>> ls(cache)
    d  dist
    d  newdir

Using relative paths
--------------------

You can use a relative path for ``download-cache`` (the same logic is applied to
``eggs-directory`` and to ``extends-cache`` too) and in such case it is considered
relative to the location of the configuration file that sets its value.

As an example, we create a ``base.cfg`` configuration in a different directory::

    >>> basedir = tmpdir('basecfg')
    >>> write(basedir, 'base.cfg',
    ... '''
    ... [buildout]
    ... download-cache = cache
    ... ''')

and a ``buildout.cfg`` that extends from there::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = %(basedir)s/base.cfg
    ... parts =
    ... ''' % globals())

    >>> dummy = system(buildout)
    >>> ls(basedir)
    -  base.cfg
    d  cache

Of course this cannot be used when the base configuration is not on the local
filesystem because it wouldn't make any sense having a remote cache::

    >>> server_data = tmpdir('server_data')
    >>> server_url = start_server(server_data)
    >>> cd(sample_buildout)

    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... download-cache = cache
    ... """)

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = %(server_url)s/base.cfg
    ... parts =
    ... ''' % globals())

    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    While:
      Initializing.
    Error: Setting "download-cache" to a non absolute location ("cache") within a
    remote configuration file...

Though, you can create the ``download-cache`` within a nested directory, so that you can
group all your generated directories (like ``eggs-directory`` or ``extends-cache`` too)
within a single directory:

    >>> test_nested = tmpdir('test_nested')
    >>> cd(test_nested)
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... download-cache = ${buildout:directory}/var/cache
    ... eggs-directory = ${buildout:directory}/var/eggs
    ... parts-directory = ${buildout:directory}/var/parts
    ... develop-eggs-directory = ${buildout:directory}/var/develop-eggs
    ... ''')

    >>> dummy = system(buildout)
    >>> ls(test_nested)
    d  bin
    -  buildout.cfg
    d  var
    >>> ls(os.path.join(test_nested, 'var'))
    d  cache
    d  develop-eggs
    d  eggs
    d  parts

