JSON-LD Community Group Telecon

Minutes for 2012-08-21

  1. DrupalCon discussion on JSON-LD
  2. Property Generators and .compact() API
  3. Alternative to combinatorial .compact()
  1. The group is committed to support language maps and property generators in JSON-LD 1.0.
  2. Support optional features in .compact(). Any optional feature that is implemented, MUST be implemented in a specific way as outlined in the JSON-LD API.
Manu Sporny
Gregg Kellogg
Gregg Kellogg, Manu Sporny, Markus Lanthaler, Niklas Lindström, François Daoust, Dave Longley
Audio Log
Gregg Kellogg is scribing.
Gregg describes talk he's giving at NoSQL Now!, slides are above, good feedback from folks.

Topic: DrupalCon discussion on JSON-LD

Manu Sporny: Lin has asked what JSON-LD features should be used in Drupal.
… Two issues that were important have been re-opened: property generators and language map.
… When they were re-opened they became nervous about stability of JSON-LD.
… We need to discuss why they were re-opened, and what to do going forward.
… The message is that the features are unstable, and could go away, which is not true.
Markus Lanthaler: The main reason they were opened was because the algorithms haven't been updated, and we haven't discussed how compaction works.
… In the property generators issue, there was a comment that this was supported, but the array-position wasn't.
… I thought we might want to discuss the array-position issue again.
… The main reason was to make sure that the algorithms are updated before the issue is closed.
Markus Lanthaler: I had some question about our support for property generators.
Manu Sporny: Perhaps we should discuss how compaction should work.
… the main thing is to signal to Drupal that we're not pulling the rug out from under them.
… We re-opened to make sure that all the work was completed, not that the feature was in doubt.
Manu Sporny: It's not just for drupal, we have a similar mechanism in RDFa, and other companies have indicated their need for it too.
… It's not just for drupal, but they did provide motivation.
Niklas Lindström: I understand that the mutliple-property feature could be seen as "un-pure" from a theoretical perspective. You should rely on inference and other things, but those are all ideals.
… This case supports a fairly easy way for both producers and consumers to manage such properties.
… The alternative would be to create different documents for different consumers, which isn't very useful.
… It's un-pure, but useful.
Manu Sporny: I don't hear any objection to keeping the feature, we just need to work out details, specifically in compaction.
Gregg Kellogg: We may want to say right now that it's not addressed in compaction, it would allow us to close out this particular feature. [scribe assist by Manu Sporny]
Gregg Kellogg: We could say that if we /do not/ describe anything happening in compaction, then we may not do anything. [scribe assist by Manu Sporny]
Gregg Kellogg: The algorithm for compaction would be a bit different... more heavyweight. [scribe assist by Manu Sporny]
Gregg Kellogg: I don't think we want to wrap these things together, we don't want to message that just because we haven't wrapped some of the API details up, that we're not going to support the feature. [scribe assist by Manu Sporny]
Niklas Lindström: +1 for that
Manu Sporny: we do a proposal that says "support both property-generators and language-maps; close issues; create new issues for compaction implications".
Markus Lanthaler: you can't make use of these things without the API, so we need all parts.
Gregg Kellogg: I think we need to do the expansion... no controversy there, the issue comes in via compaction and in via framing... certainly, there is some open work to be done in the expansion algorithms. It's reasonable to link them to the existing issues. [scribe assist by Manu Sporny]
Gregg Kellogg: Our practice has been to keep issues open until the language has been updated to reflect what's done. It's a book-keeping exercise. [scribe assist by Manu Sporny]
Manu Sporny: we need to do something to tell them that there is a clear effort to support these features.
… The proposal is to underscore that both property generators and language maps are supported features, and the group is working out the details of how the API supports them.
Manu Sporny: we could post on drupal that we're supporting it, the issue is resolved, but the work's not all done.
Manu Sporny: the reason we need to be more careful on how we use the issue tracker is because the the perception on outside groups.
PROPOSAL: The group is committed to support language maps and property generators in JSON-LD 1.0.
Gregg Kellogg: +1
Manu Sporny: +1
Niklas Lindström: +1
Markus Lanthaler: +1
François Daoust: +1
RESOLUTION: The group is committed to support language maps and property generators in JSON-LD 1.0.

Topic: Property Generators and .compact() API

Manu Sporny: Markus' point is that we've typically dealt with things all in one issue, why is this different.
Gregg Kellogg: I don't know where other people are wrt to this. The .compact() algorithm is getting to be almost unimplementably complex. [scribe assist by Manu Sporny]
Gregg Kellogg: It becomes almost unacceptable in an algorithm, when in-fact, most people typically generate JSON-LD through their own customized mechanisms. I do make use of compaction, but mostly as a way of doing fairly standard normalizations. [scribe assist by Manu Sporny]
Gregg Kellogg: I usually generate properties as arrays - so for example, dropping single value arrays. Making use of CURIEs, or other terms. I find compaction() useful for that stuff. I don't find .frame() useful at all. I usually do in-memory linking. [scribe assist by Manu Sporny]
Gregg Kellogg: It would be too much work to get the API to do what I wanted... noticed this in .frame(). If it doesn't work, it doesn't work and there is no insight into why. Very hard to debug frames. [scribe assist by Manu Sporny]
Gregg Kellogg: I think these things make it fairly useless, adding more complexity to .compact() is not something I'd like to do. [scribe assist by Manu Sporny]
Gregg Kellogg: Which term is selected for a specific property is one of those things. [scribe assist by Manu Sporny]
Gregg Kellogg: The ranking algorithm... the very act of splitting properties among multiple terms, is strange. [scribe assist by Manu Sporny]
Gregg Kellogg: if you have a property that has an equivalent IRI, the datatypes are split among those terms.... if you are trying to be entirely symmetric between compaction and expansion. It leads to unexpected results. [scribe assist by Manu Sporny]
Manu Sporny: At a high level, I agree on the complexity argument. That's one of the reasons in the beginning we said that when we talk about compaction, we leave a lot of the magic up the the implementation.
… Maybe we should define a core-set of interoperability that you should expect from compaction, but it leaves serious concerns about interoperability.
Manu Sporny: 2 questions: 1) do we expect compaction to be interoperable. 2) were do we draw the line on interoperability?
… property generators are a case where we may not want the algorithm to do anything when it goes to compact those values.
Niklas Lindström: I also agree on a high level, especially w.r.t. property generators.
… I do see the need for ranking, and tend to think that that should be interoperable.
… The mapping from RDF to elastic search indexing of data. I'd like that to be as automatic as possible.
… From my point of view, although I agree we should debate, I feel we should draw the line at where you cannot determine what term is used from the value alone.
… That rules out property generators. Does not rule out ranking, and might not language maps, but I agree that it's on the edge.
Manu Sporny: would we be opposed to some implementations implementing it, and others not.
Gregg Kellogg: In RDFa, we have examples of optional features - implementing the feature is not required, but if it is required, it has interoperability requirements. Processor graphs is one such example. [scribe assist by Manu Sporny]
Gregg Kellogg: Maybe we have a flag that allows best-effort compaction... maybe we can have a set of parameters that are associated with these optional features. Processors can declare whether they are supported or not. They're supported if the appropriate options are passed to the algorithm. It also makes testing easier. It can also reduce the computational complexity to support odd cases that are more likely to never come about. [scribe assist by Manu Sporny]
Gregg Kellogg: In that case, maybe term selection is not optional, but it is a feature that is controlled through an API flag. [scribe assist by Manu Sporny]
Niklas Lindström: I think flags are good for optional features. It should be possible to query the implementation via a .hasFeature() method. [scribe assist by Manu Sporny]
Niklas Lindström: flags are good for optional features. We should have a "has feature" method, so you can take appropriate measure.
Manu Sporny: +1 to niklasl
Manu Sporny: I agree with the general direction of the discussion. We're discussing two things:
… 1) the compact call will have a mechanism to turn off or on different features.
… 2) if a processor implements a feature, it must be done in an interoperable way.
… This would indicate that compaction for property generators would not be part of the base API, but people could implement it by providing a feature driven by a flag.
Markus Lanthaler: I'm wondering if, in this case, it might not be too difficult to support.
… It's difficult at the moment to distinguish between prefixes and terms, because every term can be a prefix. You could then have a prefix which would expand compact IRIs.
… if we limit other options on terms that are property generators, we could simplify compaction.
Manu Sporny: this would modify the feature because of complexity, and give poor signals to developers.
Markus Lanthaler: we currently silently ignore all other definitions for a prefix if it's also a term.
… I ran across some issues where a term was used as a prefix, or the other way, and other attributes of the term were not used.
Niklas Lindström: we could actually say that things define with an expanded term definition are never used as prefixes.
Markus Lanthaler: then I would need to keep track what's a prefix or what's not.
Gregg Kellogg: We might say that if it uses an expanded definition, it only uses the @id key - if that, then it can be used as a prefix. [scribe assist by Manu Sporny]
Gregg Kellogg: A term can be used as a term if it matches. [scribe assist by Manu Sporny]
Manu Sporny: If we require a : at the end of a prefix this creates a micro-syntax.
… This would make it work differently in the body of a document. The way we have it now, you need to understand that these terms would not be used as prefixes, but that's a detail developers need to worry about.
… If we have something with a ':' in it, the general case is that they would need to educate themselves about this.
… If we require a ':', it could be error prone. A single-character mistake could mess them up.
… I don't like this for those reasons, and I think developers would have a hard time learning about this.
Niklas Lindström: .. "foaf": {"@id": "…", "@prefix": true}
Gregg Kellogg: What are the issues where this happens in common practice? The only time this becomes an issue is in something like an OWL definition. [scribe assist by Manu Sporny]
Gregg Kellogg: There are very few cases here where this happens. [scribe assist by Manu Sporny]
Niklas Lindström: there were some issue when I did some RDFa/JSON-LD conversion; it's not clear from the context if I can use them for prefixes.
Gregg Kellogg: Maybe converting from RDFa to JSON-LD... [scribe assist by Manu Sporny]
Manu Sporny: the issue was how we would support property generator in compaction.
Manu Sporny: do we think that there may be optional features in compaction?
… If there optional features, we are going to say if you implement it, it has conformance requirements to guarantee interoperability.
… If a feature is optional, it does not need to be implemented, but if you do, it must conform.
… In practice, you know if the feature is there. It's only when you have a modular system that this is useful.
Markus Lanthaler: In General, I don't like this, but there may be some cases where it's useful.
… I think the cost to processor implementors is small compared to consumer's needs.
Gregg Kellogg: There is a computational concern, if these features are enabled by default, it could really slow down your workflow. [scribe assist by Manu Sporny]
Gregg Kellogg: I have to do a combinatorial search over those properties for things like property generators... that is a computational burden. [scribe assist by Manu Sporny]
Gregg Kellogg: That's for somebody that doesn't even want property generator compaction. [scribe assist by Manu Sporny]
Gregg Kellogg: Perhaps these are features where the defaults can be defined by the presence of the feature. [scribe assist by Manu Sporny]
Manu Sporny: the more I think about it, the more I'm concerned about the memory/computational overhead of compacting property generators.
… Drupal needs this to use core properties and be compatible with schema.org.
… The decision lies with the authors. It's not import for us to decide.
… If not you're okay with the computational issues, you can turn it on or off.
Gregg Kellogg: As a consumer of data, I don't specify a context... I use what is served to me. [scribe assist by Manu Sporny]
Markus Lanthaler: Then why are we concerned about compaction if people might not use it? [scribe assist by Manu Sporny]
Gregg Kellogg: Because people do use .compact() and we don't want the computational complexity in the default case. [scribe assist by Manu Sporny]
Niklas Lindström: I was thinking that if drupal started using a lot of properties for a given term, it could be that the consumers will react to that, and think that it's just noise. I wouldn't expect more than 3-5.
… If consumers want to consume such data, they can consume it incrementally, triple by triple.
Manu Sporny: we're going to need to do some "least-resistance" proposal to make some progress.
Manu Sporny: Proposals on the table: 1) Support optional features in .compact(), 2) Make all features required in .compact()
Markus Lanthaler: we already support optional features - optimize
Manu Sporny: if we support optional features, we're likely to support property generation is high. If all are required, property generation support is low.
Manu Sporny: +1 to 1), -1 to 2)
Gregg Kellogg: +1 for 1), -1 to 2)
François Daoust: +1 to 1), -1 to 2)
Markus Lanthaler: -1 to 1), +1 to 2) (except optimizations)
Niklas Lindström: +1 to 1), -0.5 to 2)
Dave Longley: following along in IRC ... keep in mind that those that want to use the feature (eg: drupal) might shy away from JSON-LD anyway if they think it's "optional" (the compaction side of it)
Dave Longley: it needs to be sold that a processor that they can use will implement it ... otherwise what good is it?
Dave Longley: that said, i'm ok with the optional path, just pointing out (if it hasn't been) a possible issue.
PROPOSAL: Support optional features in .compact(). Any optional feature that is implemented, MUST be implemented in a specific way as outlined in the JSON-LD API.
Gregg Kellogg: +1
Manu Sporny: +1
Markus Lanthaler: -1
François Daoust: +1
Niklas Lindström: +1
RESOLUTION: Support optional features in .compact(). Any optional feature that is implemented, MUST be implemented in a specific way as outlined in the JSON-LD API.

Topic: Alternative to combinatorial .compact()

Dave Longley: maybe a better solution would be to mark which term you want to use in the compaction, in the @context, so the compaction algorithm can just look there and pick it.
Dave Longley: that's really what i think people would want anyway ... instead of having to think through the algorithm in their minds
Dave Longley: and guess which term will be selected based on whichever combination is most efficient.
Dave Longley: {"@context": {"foo": {"compact: true", ... (other stuff here) ... }}
Dave Longley: that would make compaction super simple ... and it would do what you want.
Manu Sporny: you have a document with a property generator already in compact form.
… When it is expanded, three properties are expanded. When compaction, they don't get coalesced.
Manu Sporny: Proposals for property generators and .compact(): 1) Support property generator compaction as an optional feature, 2) Do not support compaction of property generators at all, 3) Support property generator compaction as a required feature, 4) Compact only the first value in the enumeration of the array
Gregg Kellogg: +1 to 1), +0 to 2), -1 to 3), -1 to 4)
Dave Longley: {"@context": {"foo": {"compact: true", ... (other stuff here) ... }}
Dave Longley: the other option is to be explicit in the context for what to compact.
Dave Longley: "foo": {"@id": "whatever", ... property generator ..., "compact": true}
Dave Longley: the problem has always been that you need to understand the algorithm. This allows you to be more clear and not have deep understanding.
… We could put the ranking algorithm into the context. You could be very explicit about what terms to pick.
Manu Sporny: my concern is that it's pretty complex, in that it doesn't hide any of the complexity from the developer.
… it's much more complicated than the other features.
Markus Lanthaler: if you want to be specific, then you only need to specify one.
Dave Longley: "foo": {"@id": [any match in here], "compact": true}
Dave Longley: "foo": {"@id": [any match in here], "compact": 1}
Dave Longley: "foo": {"@id": [any match in here], "compact": 5}
Dave Longley: one of the issues is knowing what term to pick.
Manu Sporny: I thought the way the algorithm was supposed to work is that every id must exist with the same value. Then and only then do you use foo.
… what we're trying to do here is say how property generators are round-tripped.
… this is a new way to do compaction.
Manu Sporny: I think dave's option is interesting, but it is tangentially related to property generator compaction.
Dave Longley: do they really care about this?
… I would think that, in the general case, I want to say that all these properties mean the same thing. If that's true, I don't think we need the extra complexity.
Dave Longley: if the intention is to generate multiple properties when you see a single key, I don't see there's an issue.
Manu Sporny: it does mean that it doesn't cleanly round-trip.
Manu Sporny: +1 to 1), -1 to 2), -1 to 3), -1 to 4), +0 to 5)
Markus Lanthaler: +0.5 to 1), -1 to 2) +1 to 3), -1 to 4)
+ 0.5 to 5)
François Daoust: +1 to 1), -1 to 2), -1 to 3), -1 to 4)
François Daoust: [also like 5)]
Manu Sporny: this could send mixed messages to the drupal community.
Niklas Lindström: +1 to 1), -0.5 to 2), -1 to 3), -1 to 4), +0 to 5)
Dave Longley: it's more like we're fighting for their feature. We should ask them how they feel about 5)
Markus Lanthaler: don't really understand 5) to be honest... the way I would implement this is to rank a term for each of it @id's in compaction and leave all the rest the same
Manu Sporny: this takes us into entailment, and we should tread carefully.
Dave Longley: +1 to 1) -1 to 2) +0.5 to 3) +0 to 4) +1 to 5)
… If we start to add ranking flags, we should be careful.
Manu Sporny: 1) seems to have the strongest support.
Dave Longley: I'm worried that people may feel this is license to not implement.
Manu Sporny: once people see an implementation (PHP), they'll be fine with it.
Markus Lanthaler: I can try an implementation of 1) to see how easy it is
Dave Longley: the only way for the algorithm to work is if you check both properties and values (for 1)
… when you expand "foo", it might expand into 3 different IRIs.
... However, if when you expand, you might have something that maps.
Markus Lanthaler: "foo": [ "a", "b" ] - foo: data - expand - a: data, b: data - compact - foo: data, data ??? Remove duplicates, round-trip OK?
Dave Longley: you could end up choosing the wrong term, and create more data.
… multiple property generators are even worse.
… you might then need the ranking algorithm.
Markus Lanthaler: we already have that problem. if you have two properties that map to the same IRI, they might not compact back the same.
Manu Sporny: maybe a callback could help.
Manu Sporny: for the drupal case, they definitely want to generate multiple properties on expansion.
Manu Sporny: we need to reach out to the drupal community about this.
… I'll touch base with Lin and Stephane about this.