[Checkins] SVN: hurry.resource/trunk/src/hurry/resource/ A big refactoring again:
Martijn Faassen
faassen at infrae.com
Tue Sep 30 13:30:10 EDT 2008
Log message for revision 91639:
A big refactoring again:
* instead of including a list of the resources that this inclusion is
rolled up by, explicitly specify inclusions that supersede other resources.
* made the consolidation algorithm for rolling up less dangerous.
Only eager_superseder resources are included even if not all
resources that are superseded are available.
* adjust code generator to work with the new spelling of rollups.
Changed:
U hurry.resource/trunk/src/hurry/resource/README.txt
U hurry.resource/trunk/src/hurry/resource/core.py
-=-
Modified: hurry.resource/trunk/src/hurry/resource/README.txt
===================================================================
--- hurry.resource/trunk/src/hurry/resource/README.txt 2008-09-30 15:29:43 UTC (rev 91638)
+++ hurry.resource/trunk/src/hurry/resource/README.txt 2008-09-30 17:30:09 UTC (rev 91639)
@@ -17,13 +17,12 @@
Inclusions may depend on other inclusions. A javascript resource may
for instance be built on top of another javascript resource. This
-means both of them should be loaded when the page display, the
-dependency before the resource that depends on it.
+means both of them should be loaded when the page displays.
Page components may actually require a certain inclusion in order to
be functional. A widget may for instance expect a particular
-Javascript library to loaded. We call this an *inclusion requirement* of
-the component.
+Javascript library to loaded. We call this an *inclusion requirement*
+of the component.
``hurry.resource`` provides a simple API to specify resource
libraries, inclusion and inclusion requirements.
@@ -310,7 +309,7 @@
[<ResourceInclusion 'k-debug.js' in library 'foo'>]
Modes can also be specified fully with a resource inclusion, which allows
-you to specify a different ``library`` and ``part_of`` argumnent::
+you to specify a different ``library`` argumnent::
>>> k2 = ResourceInclusion(foo, 'k2.js',
... debug=ResourceInclusion(foo, 'k2-debug.js'))
@@ -337,11 +336,12 @@
For performance reasons it's often useful to consolidate multiple
resources into a single, larger resource, a so-called
"rollup". Multiple javascript files could for instance be offered in a
-single, larger one. These consolidations can be specified when
-specifying the resource::
+single, larger one. These consolidations can be specified as a
+resource::
- >>> b1 = ResourceInclusion(foo, 'b1.js', rollups=['giant.js'])
- >>> b2 = ResourceInclusion(foo, 'b2.js', rollups=['giant.js'])
+ >>> b1 = ResourceInclusion(foo, 'b1.js')
+ >>> b2 = ResourceInclusion(foo, 'b2.js')
+ >>> giant = ResourceInclusion(foo, 'giant.js', supersedes=[b1, b2])
If we find multiple resources that are also part of a consolidation, the
system automatically collapses them::
@@ -353,62 +353,179 @@
>>> needed.inclusions()
[<ResourceInclusion 'giant.js' in library 'foo'>]
-Consolidation will not take place if only a single resource in a
-consolidation is present::
+The system will by default only consolidate exactly. That is, if only a single
+resource out of two is present, the consolidation will not be triggered::
>>> needed = NeededInclusions()
>>> needed.need(b1)
>>> needed.inclusions()
[<ResourceInclusion 'b1.js' in library 'foo'>]
-``rollups`` can also be expressed as a list of fully specified
-``ResourceInclusion``::
+Let's look at this with a larger consolidation of 3 resources::
- >>> b3 = ResourceInclusion(foo, 'b3.js',
- ... rollups=[ResourceInclusion(foo, 'giant.js')])
+ >>> c1 = ResourceInclusion(foo, 'c1.css')
+ >>> c2 = ResourceInclusion(foo, 'c2.css')
+ >>> c3 = ResourceInclusion(foo, 'c3.css')
+ >>> giantc = ResourceInclusion(foo, 'giantc.css', supersedes=[c1, c2, c3])
+
+It will not roll up one resource::
+
>>> needed = NeededInclusions()
- >>> needed.need(b1)
- >>> needed.need(b2)
- >>> needed.need(b3)
+ >>> needed.need(c1)
>>> needed.inclusions()
- [<ResourceInclusion 'giant.js' in library 'foo'>]
+ [<ResourceInclusion 'c1.css' in library 'foo'>]
+Neither will it roll up two resources::
+
+ >>> needed = NeededInclusions()
+ >>> needed.need(c1)
+ >>> needed.need(c2)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'c1.css' in library 'foo'>,
+ <ResourceInclusion 'c2.css' in library 'foo'>]
+
+It will however roll up three resources::
+
+ >>> needed = NeededInclusions()
+ >>> needed.need(c1)
+ >>> needed.need(c2)
+ >>> needed.need(c3)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giantc.css' in library 'foo'>]
+
+The default behavior is to play it safe: we cannot be certain that we
+do not include too much if we were to include ``giantc.css`` if only
+c1 and c2 are required. This is especially important with CSS
+libraries: if only ``c1.css`` and ``c2.css`` are to be included in a
+page, including ``giantc.css`` is not appropriate as that also
+includes the content of ``c3.css``, which might override and extend
+the behavior of ``c1.css`` and ``c2.css``.
+
+The situation is sometimes different with Javascript libraries, which
+can be written in such a way that a larger rollup will just include
+more functions, but will not actually affect page behavior. If we have
+a rollup resource that we don't mind kicking in even if part of the
+requirements have been met, we can indicate this::
+
+ >>> d1 = ResourceInclusion(foo, 'd1.js')
+ >>> d2 = ResourceInclusion(foo, 'd2.js')
+ >>> d3 = ResourceInclusion(foo, 'd3.js')
+ >>> giantd = ResourceInclusion(foo, 'giantd.js', supersedes=[d1, d2, d3],
+ ... eager_superseder=True)
+
+We will see ``giantd.js`` kick in even if we only require ``d1`` and
+``d2``::
+
+ >>> needed = NeededInclusions()
+ >>> needed.need(d1)
+ >>> needed.need(d2)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giantd.js' in library 'foo'>]
+
+In fact even if we only need a single resource the eager superseder will
+show up instead::
+
+ >>> needed = NeededInclusions()
+ >>> needed.need(d1)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giantd.js' in library 'foo'>]
+
+If there are two potential eager superseders, the biggest one will
+be taken::
+
+ >>> d4 = ResourceInclusion(foo, 'd4.js')
+ >>> giantd_bigger = ResourceInclusion(foo, 'giantd-bigger.js',
+ ... supersedes=[d1, d2, d3, d4], eager_superseder=True)
+ >>> needed = NeededInclusions()
+ >>> needed.need(d1)
+ >>> needed.need(d2)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giantd-bigger.js' in library 'foo'>]
+
+If there is a potential non-eager superseder and an eager one, the eager one
+will be taken::
+
+ >>> giantd_noneager = ResourceInclusion(foo, 'giantd-noneager.js',
+ ... supersedes=[d1, d2, d3, d4])
+ >>> needed = NeededInclusions()
+ >>> needed.need(d1)
+ >>> needed.need(d2)
+ >>> needed.need(d3)
+ >>> needed.need(d4)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giantd-bigger.js' in library 'foo'>]
+
+A resource can be part of multiple rollups. In this case the rollup
+that rolls up the most resources is used. So, if there are two
+potential non-eager superseders, the one that rolls up the most
+resources will be used::
+
+ >>> e1 = ResourceInclusion(foo, 'e1.js')
+ >>> e2 = ResourceInclusion(foo, 'e2.js')
+ >>> e3 = ResourceInclusion(foo, 'e3.js')
+ >>> giante_two = ResourceInclusion(foo, 'giante-two.js',
+ ... supersedes=[e1, e2])
+ >>> giante_three = ResourceInclusion(foo, 'giante-three.js',
+ ... supersedes=[e1, e2, e3])
+ >>> needed = NeededInclusions()
+ >>> needed.need(e1)
+ >>> needed.need(e2)
+ >>> needed.need(e3)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giante-three.js' in library 'foo'>]
+
Consolidation also works with modes::
- >>> b4 = ResourceInclusion(foo, 'b4.js',
- ... rollups=['giant.js'],
- ... debug=ResourceInclusion(foo, 'b4-debug.js',
- ... rollups=['giant-debug.js']))
+ >>> f1 = ResourceInclusion(foo, 'f1.js', debug='f1-debug.js')
+ >>> f2 = ResourceInclusion(foo, 'f2.js', debug='f2-debug.js')
+ >>> giantf = ResourceInclusion(foo, 'giantf.js', supersedes=[f1, f2],
+ ... debug='giantf-debug.js')
- >>> b5 = ResourceInclusion(foo, 'b5.js',
- ... rollups=['giant.js'],
- ... debug=ResourceInclusion(foo, 'b5-debug.js',
- ... rollups=['giant-debug.js']))
+ >>> needed = NeededInclusions()
+ >>> needed.need(f1)
+ >>> needed.need(f2)
+ >>> needed.inclusions()
+ [<ResourceInclusion 'giantf.js' in library 'foo'>]
+ >>> needed.inclusions(mode='debug')
+ [<ResourceInclusion 'giantf-debug.js' in library 'foo'>]
+What if the rolled up resources have no mode but the superseding resource
+does? In this case the superseding resource's mode has no meaning, so
+modes have no effect::
+
+ >>> g1 = ResourceInclusion(foo, 'g1.js')
+ >>> g2 = ResourceInclusion(foo, 'g2.js')
+ >>> giantg = ResourceInclusion(foo, 'giantg.js', supersedes=[g1, g2],
+ ... debug='giantg-debug.js')
>>> needed = NeededInclusions()
- >>> needed.need(b4)
- >>> needed.need(b5)
+ >>> needed.need(g1)
+ >>> needed.need(g2)
>>> needed.inclusions()
- [<ResourceInclusion 'giant.js' in library 'foo'>]
+ [<ResourceInclusion 'giantg.js' in library 'foo'>]
>>> needed.inclusions(mode='debug')
- [<ResourceInclusion 'giant-debug.js' in library 'foo'>]
+ [<ResourceInclusion 'giantg.js' in library 'foo'>]
-A resource can be part of multiple rollups. In this case the rollup that
-rolls up the most resources is used::
+What if the rolled up resources have a mode but the superseding resource
+does not? Let's look at that scenario::
- >>> b6 = ResourceInclusion(foo, 'b6.js',
- ... rollups=['giant.js', 'even_bigger.js'])
- >>> b7 = ResourceInclusion(foo, 'b7.js',
- ... rollups=['giant.js', 'even_bigger.js'])
- >>> b8 = ResourceInclusion(foo, 'b8.js',
- ... rollups=['even_bigger.js'])
+ >>> h1 = ResourceInclusion(foo, 'h1.js', debug='h1-debug.js')
+ >>> h2 = ResourceInclusion(foo, 'h2.js', debug='h2-debug.js')
+ >>> gianth = ResourceInclusion(foo, 'gianth.js', supersedes=[h1, h2])
>>> needed = NeededInclusions()
- >>> needed.need(b6)
- >>> needed.need(b7)
- >>> needed.need(b8)
+ >>> needed.need(h1)
+ >>> needed.need(h2)
>>> needed.inclusions()
- [<ResourceInclusion 'even_bigger.js' in library 'foo'>]
+ [<ResourceInclusion 'gianth.js' in library 'foo'>]
+Since there is no superseder for the debug mode, we will get the two
+resources, not rolled up::
+
+ >>> needed.inclusions(mode='debug')
+ [<ResourceInclusion 'h1-debug.js' in library 'foo'>,
+ <ResourceInclusion 'h2-debug.js' in library 'foo'>]
+
+XXX superseding along with dependencies...
+
Rendering resources
-------------------
@@ -457,40 +574,61 @@
Sometimes it is useful to generate code that expresses a complex
resource dependency structure. One example of that is in
-``hurry.yui``. We can use this to render resource inclusions::
+``hurry.yui``. We can the ``generate_cod`` function to render resource
+inclusions::
+ >>> i1 = ResourceInclusion(foo, 'i1.js')
+ >>> i2 = ResourceInclusion(foo, 'i2.js', depends=[i1])
+ >>> i3 = ResourceInclusion(foo, 'i3.js', depends=[i2])
+ >>> i4 = ResourceInclusion(foo, 'i4.js', depends=[i1])
+ >>> i5 = ResourceInclusion(foo, 'i5.js', depends=[i4, i3])
+
>>> from hurry.resource import generate_code
- >>> print generate_code(a1=a1, a2=a2, a3=a3, a4=a4, a5=a5)
+ >>> print generate_code(i1=i1, i2=i2, i3=i3, i4=i4, i5=i5)
from hurry.resource import Library, ResourceInclusion
<BLANKLINE>
foo = Library('foo')
<BLANKLINE>
- a1 = ResourceInclusion(foo, 'a1.js')
- a2 = ResourceInclusion(foo, 'a2.js', depends=[a1])
- a3 = ResourceInclusion(foo, 'a3.js', depends=[a2])
- a4 = ResourceInclusion(foo, 'a4.js', depends=[a1])
- a5 = ResourceInclusion(foo, 'a5.js', depends=[a4, a3])
+ i1 = ResourceInclusion(foo, 'i1.js')
+ i2 = ResourceInclusion(foo, 'i2.js', depends=[i1])
+ i3 = ResourceInclusion(foo, 'i3.js', depends=[i2])
+ i4 = ResourceInclusion(foo, 'i4.js', depends=[i1])
+ i5 = ResourceInclusion(foo, 'i5.js', depends=[i4, i3])
-Let's look at an example with modes and rollups::
+Let's look at a more complicated example with modes and superseders::
- >>> print generate_code(b4=b4, b5=b5)
+ >>> j1 = ResourceInclusion(foo, 'j1.js', debug='j1-debug.js')
+ >>> j2 = ResourceInclusion(foo, 'j2.js', debug='j2-debug.js')
+ >>> giantj = ResourceInclusion(foo, 'giantj.js', supersedes=[j1, j2],
+ ... debug='giantj-debug.js')
+
+ >>> print generate_code(j1=j1, j2=j2, giantj=giantj)
from hurry.resource import Library, ResourceInclusion
<BLANKLINE>
foo = Library('foo')
<BLANKLINE>
- b4 = ResourceInclusion(foo, 'b4.js', rollups=['giant.js'], debug=ResourceInclusion(foo, 'b4-debug.js', rollups=['giant-debug.js']))
- b5 = ResourceInclusion(foo, 'b5.js', rollups=['giant.js'], debug=ResourceInclusion(foo, 'b5-debug.js', rollups=['giant-debug.js']))
+ j1 = ResourceInclusion(foo, 'j1.js', debug='j1-debug.js')
+ j2 = ResourceInclusion(foo, 'j2.js', debug='j2-debug.js')
+ giantj = ResourceInclusion(foo, 'giantj.js', supersedes=[j1, j2], debug='giantj-debug.js')
We can control the name the inclusion will get in the source code by
using keyword parameters::
- >>> print generate_code(hoi=a1)
+ >>> print generate_code(hoi=i1)
from hurry.resource import Library, ResourceInclusion
<BLANKLINE>
foo = Library('foo')
<BLANKLINE>
- hoi = ResourceInclusion(foo, 'a1.js')
-
+ hoi = ResourceInclusion(foo, 'i1.js')
+
+ >>> print generate_code(hoi=i1, i2=i2)
+ from hurry.resource import Library, ResourceInclusion
+ <BLANKLINE>
+ foo = Library('foo')
+ <BLANKLINE>
+ hoi = ResourceInclusion(foo, 'i1.js')
+ i2 = ResourceInclusion(foo, 'i2.js', depends=[hoi])
+
Sorting inclusions by dependency
--------------------------------
Modified: hurry.resource/trunk/src/hurry/resource/core.py
===================================================================
--- hurry.resource/trunk/src/hurry/resource/core.py 2008-09-30 15:29:43 UTC (rev 91638)
+++ hurry.resource/trunk/src/hurry/resource/core.py 2008-09-30 17:30:09 UTC (rev 91639)
@@ -22,22 +22,24 @@
"""
implements(interfaces.IResourceInclusion)
- def __init__(self, library, relpath, depends=None, rollups=None, **kw):
+ def __init__(self, library, relpath, depends=None,
+ supersedes=None, eager_superseder=False, **kw):
"""Create a resource inclusion
- library - the library this resource is in
- relpath - the relative path from the root of the library indicating
- the actual resource
- depends - optionally, a list of resources that this resource depends
- on. Entries in the list can be
- ResourceInclusions or strings indicating the path.
- In case of a string, a ResourceInclusion assumed based
- on the same library as this inclusion.
- rollups - optionally, a list of resources that this resource can
- be rolled up into. Entries in the list can be
- ResourceInclusions or strings indicating the path.
- In case of a string, a ResourceInclusion assumed based
- on the same library as this inclusion.
+ library - the library this resource is in
+ relpath - the relative path from the root of the library indicating
+ the actual resource
+ depends - optionally, a list of resources that this resource depends
+ on. Entries in the list can be
+ ResourceInclusions or strings indicating the path.
+ In case of a string, a ResourceInclusion assumed based
+ on the same library as this inclusion.
+ supersedes - optionally, a list of resources that this resource
+ supersedes as a rollup resource. If all these
+ resources are required, the superseding resource
+ instead will show up.
+ eager_superseder - even if only part of the requirements are
+ met, supersede anyway
keyword arguments - different paths that represent the same
resource in different modes (debug, minified, etc),
or alternatively a fully specified ResourceInclusion.
@@ -48,17 +50,35 @@
assert not isinstance(depends, basestring)
depends = depends or []
self.depends = normalize_inclusions(library, depends)
-
- assert not isinstance(rollups, basestring)
- rollups = rollups or []
- self.rollups = normalize_inclusions(library, rollups)
+
+ self.rollups = []
normalized_modes = {}
for mode_name, inclusion in kw.items():
normalized_modes[mode_name] = normalize_inclusion(
library, inclusion)
self.modes = normalized_modes
+
+ assert not isinstance(supersedes, basestring)
+ self.supersedes = supersedes or []
+ self.eager_superseder = eager_superseder
+ # create a reference to the superseder in the superseded inclusion
+ for inclusion in self.supersedes:
+ inclusion.rollups.append(self)
+ # also create a reference to the superseding mode in the superseded
+ # mode
+ # XXX what if mode is full-fledged resource inclusion which lists
+ # supersedes itself?
+ for mode_name, mode in self.modes.items():
+ for inclusion in self.supersedes:
+ superseded_mode = inclusion.mode(mode_name)
+ # if there is no such mode, let's skip it
+ if superseded_mode is inclusion:
+ continue
+ mode.supersedes.append(superseded_mode)
+ superseded_mode.rollups.append(mode)
+
def __repr__(self):
return "<ResourceInclusion '%s' in library '%s'>" % (
self.relpath, self.library.name)
@@ -151,31 +171,38 @@
return result
def consolidate(inclusions):
- # first map rollup -> list of inclusions that are in this rollup
- rollup_to_inclusions = {}
+ # keep track of rollups: rollup key -> set of inclusion keys
+ potential_rollups = {}
for inclusion in inclusions:
for rollup in inclusion.rollups:
- rollup_to_inclusions.setdefault(rollup.key(), []).append(
- inclusion)
+ s = potential_rollups.setdefault(rollup.key(), set())
+ s.add(inclusion.key())
- # now replace inclusion with rollup consolidated biggest amount of
- # inclusions, or keep inclusion if no such rollup exists
+ # now go through inclusions, replacing them with rollups if
+ # conditions match
result = []
for inclusion in inclusions:
- potential_rollups = []
+ eager_superseders = []
+ exact_superseders = []
for rollup in inclusion.rollups:
- potential_rollups.append((len(rollup_to_inclusions[rollup.key()]),
- rollup))
- if not potential_rollups:
- # no rollups at all
- result.append(inclusion)
- continue
- sorted_rollups = sorted(potential_rollups)
- amount, rollup = sorted_rollups[-1]
- if amount > 1:
- result.append(rollup)
+ s = potential_rollups[rollup.key()]
+ if rollup.eager_superseder:
+ eager_superseders.append(rollup)
+ if len(s) == len(rollup.supersedes):
+ exact_superseders.append(rollup)
+ if eager_superseders:
+ # use the eager superseder that rolls up the most
+ eager_superseders = sorted(eager_superseders,
+ key=lambda i: len(i.supersedes))
+ result.append(eager_superseders[-1])
+ elif exact_superseders:
+ # use the exact superseder that rolls up the most
+ exact_superseders = sorted(exact_superseders,
+ key=lambda i: len(i.supersedes))
+ result.append(exact_superseders[-1])
else:
- result.append(inclusion)
+ # nothing to supersede resource so use it directly
+ result.append(inclusion)
return result
def sort_inclusions_by_extension(inclusions):
@@ -186,11 +213,7 @@
return sorted(inclusions, key=key)
def sort_inclusions_topological(inclusions):
- """Sort inclusions by dependency.
-
- Note that this is not actually used in the system, but can be used
- to resort inclusions in case sorting order is lost - or if the
- assumptions in this library turn out to be incorrect.
+ """Sort inclusions by dependency and supersedes.
"""
dead = {}
result = []
@@ -207,6 +230,8 @@
dead[inclusion.key()] = True
for depend in inclusion.depends:
_visit(depend, result, dead)
+ for depend in inclusion.supersedes:
+ _visit(depend ,result, dead)
result.append(inclusion)
def render_css(url):
@@ -271,8 +296,10 @@
depends_s = ', depends=[%s]' % ', '.join(
[inclusion_to_name[d.key()] for d in inclusion.depends])
s += depends_s
- if inclusion.rollups:
- s += ', ' + _generate_inline_rollups(inclusion)
+ if inclusion.supersedes:
+ supersedes_s = ', supersedes=[%s]' % ', '.join(
+ [inclusion_to_name[i.key()] for i in inclusion.supersedes])
+ s += supersedes_s
if inclusion.modes:
items = []
for mode_name, mode in inclusion.modes.items():
@@ -287,39 +314,9 @@
return '\n'.join(result)
def generate_inline_inclusion(inclusion, associated_inclusion):
- if (inclusion.library.name == associated_inclusion.library.name and
- not inclusion.rollups):
+ if inclusion.library.name == associated_inclusion.library.name:
return "'%s'" % inclusion.relpath
else:
- s = "ResourceInclusion(%s, '%s'" % (inclusion.library.name,
- inclusion.relpath)
- if inclusion.rollups:
- s += ', ' + _generate_inline_rollups(inclusion)
- s += ')'
- return s
-
-def _generate_inline_rollups(inclusion):
- return 'rollups=[%s]' % ', '.join(
- [generate_inline_inclusion(r, inclusion)
- for r in inclusion.rollups])
-
-def generate_inclusion_name(inclusion, used_names):
- rest, fullname = os.path.split(inclusion.relpath)
- name, ext = os.path.splitext(fullname)
- if name not in used_names:
- used_names.add(name)
- return name
- name = name + ext
- name = name.replace('.', '_')
- if name not in used_names:
- used_names.add(name)
- return name
- i = 0
- while True:
- name = name + str(i)
- if name not in used_names:
- used_names.add(name)
- return name
- assert False, "Not possible to generate a unique name!"
-
+ return "ResourceInclusion(%s, '%s')" % (inclusion.library.name,
+ inclusion.relpath)
More information about the Checkins
mailing list