JSON Pointer spec (RFC 6901) support

[markkolich]

Consider the following JSON object:

{
  "foo": {
    "bar": {
       "baz": "bat"
    }
  }
}

Using GSON, without registering a custom type adapter, if I want to safely access member foo.bar.baz I end up with something like this:

final JsonObject object = new GsonBuilder().create().fromJson("{...}", JsonObject.class);

final JsonObject foo = object.getAsJsonObject("foo");
if (foo != null) {
  final JsonObject bar = foo.getAsJsonObject("bar");
  if (bar != null) {
    final JsonElement baz = bar.get("baz");
    if (baz != null && baz.isJsonPrimitive()) {
      final String bat = baz.getAsString();
      // ...
    }
  }
}

Instead, I'd love to be able to call a single method on the initial JsonObject itself, something like getStringAtPath():

final JsonObject object = new GsonBuilder().create().fromJson("{...}", JsonObject.class);

final String baz = object.getStringAtPath("foo.bar.baz"); // <<-- !!
if (baz == null) {
  // ... didn't exist.
}

In lieu of this missing API, I cooked up a simple helper method for String primitives:

import com.google.common.base.Preconditions;
import com.google.common.base.Splitter;

@Nullable
public static final String getStringAtPath(
    final JsonObject object,
    final String path) {
  Preconditions.checkNotNull(object, "JSON object cannot be null.");
  Preconditions.checkNotNull(path, "JSON object path cannot be null.");

  JsonObject base = object;
  final Iterable<String> segments = Splitter.on(".").split(path);
  for (final String segment : segments) {
    final JsonElement element = base.get(segment);
    if (element == null) {
      return null;
    }

    if (element.isJsonPrimitive()) {
      final JsonPrimitive primitive = element.getAsJsonPrimitive();
      return primitive.getAsString();
    } else if (element.isJsonObject()) {
      base = element.getAsJsonObject();
    } else {
      return null;
    }
  }

  return null;
}

It would be tremendously helpful if similar methods were conveniently folded into the JsonObject API itself. It could easily be extended for the other types as well: int, boolean, char, double, etc. Admittedly, things get a bit hairy with arrays but I'd imagine support for paths like foo.bar[8].baz could work.

Apologies if this is a dupe, I did briefly search the existing set of issues but didn't immediately find anything related.

[lyubomyr-shaydariv]

I think Gson will never have something like that, but there are tools like JsonPath that seems to be exactly what you're looking for. It provides a powerful generic query language providing support for multiple libraries including Gson.

[markkolich]

there are tools like JsonPath that seems to be exactly what you're looking for.

That's certainly an option, but I was hoping to avoid pulling in yet another library for something that seems so common and fundamental.

In fact, Jackson 2.3.0 introduced support for the JSON Pointer spec (RFC 6901), which is closer to what I'm looking for:

final JsonNode object = new ObjectMapper().readTree("{...}");
final String baz = object.at("/foo/bar/baz").asText();

I'm heavily invested in GSON, and have no intention of switching to Jackson just for JSON Pointer support. However, I was hoping GSON would follow Jackson's lead on this.

[scottslewis]

We've just learned of this enhancement request.

There is an existing proposal called local only JRef and this has been implemented in java and python as well as ts/js.

We would be willing to contribute all or parts of these implementations to gson to provide

1. Smaller/more efficient transmission of object graph serialization ($refs rather than copies)

2. Interoperability with other languages/libraries via compliance with the rfc 6901 spec

[eamonnmcmanus]

It doesn't seem that we should need to modify Gson itself to support JSON Pointer. That could easily be implemented as a separate library on top of the Gson API.

On the other hand, JSON Pointer is a standard (RFC 6901), and it does provide an alternative to the often clumsy way you need to code access to nested members. I think we'd be OK with a PR that extended the JsonElement class with a method something like JsonElement at(String path). We would want decent unit test coverage, of course.

JRef, on the other hand, is not a standard I believe. It is also a fair bit more complicated, even with a local-only subset. I think supporting it would be better done via a separate library. If such a library existed, we could reference it in the Gson documentation.

[scottslewis]

It doesn't seem that we should need to modify Gson itself to support JSON Pointer. That could easily be implemented as a separate library on top of the Gson API.

On the other hand, JSON Pointer is a standard (RFC 6901), and it does provide an alternative to the often clumsy way you need to code access to nested members. I think we'd be OK with a PR that extended the JsonElement class with a method something like JsonElement at(String path). We would want decent unit test coverage, of course.

Ok. I'm willing to do this (including tests), if the gson team can commit to some further assistance with code guidance, expectations wrt tests, project-specific conventions around releng, ci and whatever.

JRef, on the other hand, is not a standard I believe. It is also a fair bit more complicated, even with a local-only subset.

JRef is not a standard...we are not claiming otherwise. Jason and I are doing what we can to change that, but apparently the IETF json standardization people (e.g. 6901) have mostly spread to the winds and/or moved onto (e.g.) json schema and/or other 'greener pastures'. This creates an adoption/standardization chicken-and-egg problem.

If you have some influence or connections wrt full standardization we would be happy to work with you and/or others.

I do want to emphasize though that as you can tell by examining any of the implementations I referenced that the local-only JRef is really just the url fragment syntax specified in rfc 6901. Without the full uri-support (local-only) there is nothing more to it.

I think supporting it would be better done via a separate library.

Why?

If such a library existed, we could
reference it in the Gson documentation.

Ok. I/we are willing to consider this, but as I'm sure you know maintenance of libraries can be costly (I'm a sole maintainer myself of a > 20 years, heavily-consumed library). The JRef local only impls are exceedingly small and simple. IMHO, so small and simple (and incomplete without ser library) that it could be difficult to get developer visibility.

Still, we have made a proposal for all repos at OpenMCPTools including JRef to be a AAIF (agentic ai foundation) project as well. I'm hopeful that this could allow us to pursue the cross-lib/language interoperability/standardization aspect across libraries and languages.

[Marcono1234]

I think we'd be OK with a PR that extended the JsonElement class with a method something like JsonElement at(String path)

In case someone creates a PR, it would probably be useful to a look at how Jackson supports JSON Pointers (and inspecting their open issues and feature requests), to learn about potential pitfalls in advance before deciding on an API and implementation.

Also, if such an implementation is based on the JSON Pointer RFC, then it would probably be good to refer to it in Gson's API and documentation only as "JSON Pointer" / "Pointer" (and not "path"), because recently JSONPath was standardized as well.

(Just my personal opinion on this though.)

[scottslewis]

I think we'd be OK with a PR that extended the JsonElement class with a method something like JsonElement at(String path)

In case someone creates a PR, it would probably be useful to a look at how Jackson supports JSON Pointers (and inspecting their open issues and feature requests), to learn about potential pitfalls in advance before deciding on an API and implementation.

Agreed. Please see Jackson issue. Also see longer discussion

Also, if such an implementation is based on the JSON Pointer RFC, then it would probably be good to refer to it in Gson's API and documentation only as "JSON Pointer" / "Pointer" (and not "path"), because recently JSONPath was standardized as well.

Also agreed.

To be very clear: The implementations at jref_java, jref_python, jref_rust, and jref_typescript are intended to only aid integration of local-only json ptr/6901 support with all/any existing impls/libs in all languages.

JRef is not and will never be functionally competitive with any existing json serialization libraries. Actually the serialization functions in the jref src don't do actual serialization at all, but rather only do reference creation and resolve using the 6901 specified syntax.

Rather, it's our intention and desire to have these impls serve as something of a simple reference impl...for cross-language/lib interoperability and hopefully eventual formal standardization.

I would/will be happy to create a pr with the jref_java serialize/deserialize, and will do so if asked. But my own experience as a long-time maintainer of a standards-based API tells me that probably won't work well since I don't have enough information about the internals of gson or any of the existing serialization libraries.

Thus this issue+comments.

[scottslewis]

There is a pr to Jackson for supporting json pointers. The change is a single JRefModule.java file using Jackson's module system and ObjectMapper to serialize repeat references in object graph as json pointers and deserialize and resolve json pointers to reproduce the same object graph on receiver.

I would like to make a similar contribution to Gson, probably using Gson's type adapter factory.

A large part of the Jackson contribution was vastly simplified by the presence in Jackson core of the JsonPointer class. I'm not able to find in current Gson anything akin to the JsonPointer class. If there is by some other name or somewhere else, please direct me.

Unless not needed, I will begin work on a JsonPointer implementation for Gson...with changes of course for the use of JsonReader rather than Jackson's types.

Question: is Java-8 still the minimum execution environment for Gson? Or will it be something higher?

Also, when is the next anticipated release of Gson?

[scottslewis]

I've created a JRefTypeAdapter factory and tests in this project. If a pr would be accepted, I will open one.

The only change/addition to existing Gson (other than the JRefTypeAdapterFactory) is to add the following method to JsonWriter

  public String getDeferredName() {
	  return this.deferredName;
  }

[scottslewis]

It doesn't seem that we should need to modify Gson itself to support JSON Pointer. That could easily be implemented as a separate library on top of the Gson API.

On the other hand, JSON Pointer is a standard (RFC 6901), and it does provide an alternative to the often clumsy way you need to code access to nested members. I think we'd be OK with a PR that extended the JsonElement class with a method something like JsonElement at(String path). We would want decent unit test coverage, of course.

JRef, on the other hand, is not a standard I believe. It is also a fair bit more complicated, even with a local-only subset.
I think supporting it would be better done via a separate library. If such a library existed, we could reference it in the Gson documentation.

Such a library now exists

I've also offered to contribute this impl to Gson in pr #3050.

IMO this doesn't make sense as a separate library. The reason: all of the json pointer/standardized bits are implemented by the Jackson JsonPointer and TokenStreamContext classes (included with Jackson team permission as described in the pr) and all of the rest of this code is using/implementing Gson's TypeAdapterFactory/TypeAdapter extension mechanism. The JRef-specific bit is literally the use of a single string constant "$ref". The code in JRefTypeAdapterFactory/JRefTypeAdapter is for connecting the JsonPointer spec (rfc-6901) to gson's use of json path for producing a JsonPointer to represent > 1 references in the local object graph.

And FWIW, Jason Derosier's (JRef spec author) has told me that the json schema standards work is moving forward wrt the IETF specs (like rfc-6901). The standards-body jurisdiction sounds to me like the complicated part of things wrt jref and standardization of the serialized JsonPointer name.

I agree what I've contributed here is the equivalent to a 'feature' for json ser/des...i.e. something the use case dictates as necessary or not. For the Jackson version, it's implemented as a 'jackson module'.

Again I don't think it makes sense as a separate library. It's a very small amount of code that's completely dependent upon Gson's extension mechanisms, and would be very difficult to maintain and distribute as a separate library because of that dependence on Gson's TypeAdapterFactory.

[scottslewis]

Would a project committer please review pr #3050?

[scottslewis]

Please see #3050 (comment)

[scottslewis]

It doesn't seem that we should need to modify Gson itself to support JSON Pointer. That could easily be implemented as a separate library on top of the Gson API.

This is what I've done in pr #3050. The impl uses JRefTypeAdapter api to implement json ptr creation and parsing via jref.

On the other hand, JSON Pointer is a standard (RFC 6901), and it does provide an alternative to the often clumsy way you need to code access to nested members. I think we'd be OK with a PR that extended the JsonElement class with a method something like JsonElement at(String path). We would want decent unit test coverage, of course.

As it turns out using JRefTypeAdapter makes it unnecessary to add to JsonElement. Rather, the impl dynamically converts a json path to a json pointer. Only one new...internal,not api...method is required.

JRef, on the other hand, is not a standard I believe. It is also a fair bit more complicated, even with a local-only subset.

WRT the implementation in #3050, jref adds just one thing to json pointer spec/syntax. That's the name of the json pointer property ("$ref") in serialized reference objects...e.g. '{ "$ref": "#path/to/local/object/1" }'. Everything else is specified by the json pointer spec (pointer syntax).

I think supporting it would be better done via a separate library. If such a library existed, we could reference it in the Gson documentation.

I have made it #3050 available as a separate library https://github.com/OpenMCPTools/jref-lib/tree/main/java/jref-gson

As the impl is very small (< 5 classes), requires/only works with GSon JRefTypeAdapter api, and doesn't do anything else, it seems to me more like a GSon feature/enhancement than a separate library. Thus #3050.