shepard Architectural Documentation

Collections are meant as the root or container of your entire structure. All information, data, files, permissions, etc. are related to exactly one collection. Usually, a collection is the first thing you create.

In order to organize your project, you use Data Objects. They are very generic and can be used to organize your work as you wish. You can use them to create a tree based on experiments, lots, process steps or whatever you want.

There are different types of container available, e.g. for files, structured data, timeseries or semantic repositories. You can use them to group things together. If you want to store some images of a video camera, you can create a file container and upload them. If you need to store coordinates of a robot movement, you can create a timeseries containers and store the data there.

The Lab Journal enables researchers to document their experiments, data and results. Lab Journal entries can be linked to any Data Object. They also support html content, so that basic formatting and structuring can be used.

The backend is split into modules by their functionality. That means that there is e.g. a module for timeseries including managing containers and timeseries data, a module for managing files, for collections & data objects etc.

Each of those modules contains all technical components needed to fulfill it’s purpose, including endpoints, services, domain model objects, DAOs, entities and repositories.

Additionally, there may be modules for special functionalities like authentication.

See Backend Level 2 for the concrete modularization.

We use quarkus-rest to define REST endpoints.

For authentication and general request validation, we use a combination of filters following the Jakarta REST way and checks in the backend services. The Jakarta filters can be found at filters/. In general, all requests need to be done by authenticated users. The JWTFilter and the UserFilter take care of validating authentication. The actual authentication checks, i.e., if a user is allowed to access a resource, are executed in the backend’s service layer.

Some endpoints should be public, for example /healthz and /versionz. To make this possible we use the PublicEndpointRegistry class. In this class we register all public endpoints in a static string array. The authentication filters will be bypassed for endpoints in this array. Since the /healthz endpoint is automatically public thanks to the SmallRye Health extension, we don’t need to add it to the PublicEndpointRegistry.

Each route is defined in pages/. The root file of a route itself should not contain a lot of logic, instead it should invoke one or more components.

In components/ all components are stored. These are grouped in folders by domain, e.g. components/collections.

Stateful logic can be extracted into composables and are stored in composables/.

Stateless utility functions should be stored under utils/.

For the interaction with the backend we check a generated openapi client into the repository, see here for more information.

In order to directly instantiate the API clients with our default configuration, we use the createApiInstance utility.

The current LTS version of Node.js should be used in all of the project (Gitlab jobs, docker images, local development, etc.) to maintain reliability and performance.

The current LTS version of Quarkus should be used in our backend to maintain reliability and performance.

Contained Blackboxes:

Contained Blackboxes:

The Lab Journal module allows users to create, edit and delete journal entries. Lab Journal Entries can be used for documentation purposes. This feature is thought to be used via the frontend but can also be used directly via the REST interface if needed. Lab Journal entries are stored in the neo4j database. This module has a dependency to the Collections & DataObjects module because they are always linked to a DataObject. This module also has a dependency to the Authoriziation module because the user needs the correct permissions to see and edit lab journal entries.

Following our solution strategy, we have the following classes:

This module handles persisting timeseries data in a TimescaleDB. It includes all relevant endpoints and services. The database schema includes two tables:

The schema for both tables is defined by the database migrations in src/main/java/resources/db/migration. The timeseries table is managed in the code using hibernate entities. The data point table is managed directly using custom queries, since we want to make full use of TimescaleDB features and performance.

This module provides functionality to manage spatial data. In order to work with spatial data, a SpatialDataContainer has to be created. This container allows users to upload spatial data points with three dimensions. The spatial data is stored in a PostGIS database. The module provides REST endpoints to interact with the spatial data. Especially the GET endpoints are very useful because they make use of special PostGIS functions like retrieving data near a point or within a bounding box or sphere.

The following artifacts are provided as documentation of shepard:

For interaction with Neo4j we use neo4j-ogm. The connection is defined in src/main/java/de/dlr/shepard/common/neo4j/NeoConnector.java. The packages of all entities have to be registered in the SessionFactory there in order to work properly.

In addition to OIDC, we also allow authentication via API keys. Shepard generates these keys itself and stores them in our internal database. Although the API keys are also JWTs, we have to check whether the specified key can be found in our database. Otherwise, we would continue to accept keys that have already been deleted, which is not the intended behavior.

Authentication data can be accessed in backend services using the AuthenticationContext. It is request scoped and the data is populated in the JWTFilter.

It can be injected using dependency injection.

Almost all request to the backend are checked when they try to access resources like a Collection. These checks are implemented in the service layer of the backend and can therefore be found in almost any *Service.java file. An example of one such check can be seen in the deleteContainer function of the FileContainerService.java file:

Before the container is deleted, the existence of the FileContainer is checked with the getContainer method. This method throws a NotFoundException/InvalidPathException if the container could not be found. It should be assumed that all service layer function throw either a NotFoundException or an`InvalidAuthException`. Only some service layer functions return null or Optional<> when encountering errors. It is important to run the existence check before the permission check, to make sure that the object is actually there before accessing it’s permissions.

After this existence check, the getContainer method also checks that the currently requesting user has READ permissions on the object. If this is not the case, an InvalidAuthException is thrown. After calling the getContainer method, an assertion (assertIsAllowedToDeleteContainer) is executed to check if the user has the correct permissions to delete the container. If this assertion fails, an InvalidAuthException is thrown. Only when both of these checks for existence and READ/ WRITE permissions succeed, the FileService deletes the container.

Almost all service classes implement this check in an analogous way. They provide a getABC() method, that checks for existence and READ permissions, and on success, returns the object. Furthermore, they have access to an assert function that checks for READ, WRITE, and MANAGE permissions. Often, the getABC() function is only utilized to check for the existence and READ permissions of the object, without actually using the returned object. This check structure provides a fine-grained permission and validity control on the backend service level.

The described approach above, to have a permission check in almost all service layer methods, leads to the problem that many database lookups have to be done, to check the permissions object every time a resource is requested. Since, the service layer functions are sometimes called in a cascading style, many of these permission lookups are done. To prevent this, we introduced a LastSeenCache system. There are three different LastSeenCaches:

Here is an example of how the PermissionsLastSeenCache is used in a function of the PermissionService to check if a user has the correct permission to access an entity:

Before running checks to determine if the user is allowed to access the entity, a fast cache-lookup is done. The key is constructed by creating a string from the entityId, the access type and the username. If the key is present, this function directly returns true. Else, the isAllowed check is run, and if the user is allowed to access the entity, this information is stored in the cache.

Each cache entry is saved with a Date that stores the information when the cache item was added. After a certain amount of time, the cache entry expires and is no longer valid. All LastSeenCaches can be configured to use different values for this expiration time. By default it is 5 minutes.

To be able to authenticate in the frontend and acquire a JWT token, we use @sidebase/nuxt-auth as our authentication module in the frontend.

We handle the merge requests opened by renovate by performing the following steps for each update:

Also, the dependencies in package-lock.json should be updated. This is done by running npm update in the top level directory.

In case we could not perform the update it should be suspended and documented in the list of suspended updates. The reason can either be too much effort (we create a new story for that update) or that the update is not possible or feasible right now.

This can be done by excluding the library or specific version in the renovate config. Afterwards the config needs to be merged to main with the following commands:

Afterwards the merge request can be closed.

Sometimes, when the configuration changes or dependency updates were done without renovate, the bot might abandon a merge request. In this case the merge request is not automatically closed and has to be closed manually. The corresponding branch must also be deleted manually to keep things clean.

BARTOC knows about terminology registries, including itself. Registries also provide access to full terminologies either via an API (terminology service) or by other means (terminology repository).

https://bartoc.org/registries

Typical "interfaces":

(others could in include eclass or IEEE iris)

Query documents using native mongoDB mechanics

Query collections, data objects and references

Usually a new shepard version is released on the first Monday of the month. However, this date is not fixed and can be postponed by a few days if necessary. This monthly release increases the release version number.

We use semantic versioning, meaning that the version number consists of a major, minor and patch number in the following format: MAJOR.MINOR.PATCH. Minor is the default version increase for a release. Breaking changes imply a Major release. And hotfixes or patches are performed as a Patch release (see Hotfix process below).

Currently there are two workflows for releases, one minor/ major release and a patch release. Both release types are explained step by step below.

These steps describe a regular (monthly) release for shepard but can also be used to release an unplanned patch release.

Furthermore, there are two ways to create an unplanned patch/ hotfix release.

The first option is the more classical hotfix approach, meaning that it only brings the changes from merge requests containing hotfixes from the develop branch to the main branch. The steps needed for this option are explained below in the section: Performing a hotfix release.

The second option includes creating an MR containing the needed patch changes on develop branch, then merge the develop branch on main and create a new minor release. This would create a new out-of-cycle release containing the patch and all changes from develop since the last release. This option is the same procedure as a regular release, which is described right below.

Hotfixes are changes to the main branch outside the regular releases to fix urgent bugs or make small changes that need to be applied to the default branch of the project. The steps below describe how one can release a single hotfix MR without having to merge the develop branch on main. This means that the other changes on develop branch are only merged when a new regular release is created.

This section is a short summary of this page.

This section is a short summary of this page.

As a shepard user I want to be able to use different versions of data sets to facilitate collaboration in a research project and lay the groundwork for future features like branching, visualization of differences or restore functionality.

I can define a version of a collections to mark a milestone in the project in order to freeze the status that the data set has right now via the API. There is always the one active version called HEAD which is the working copy that can be edited on shepard and there can be n versions on shepard that are then read-only. If I never define a version as a user then there is no change in functionality for me. Versions are identified by a UUID. Version should be applied for organizational elements, not for payload data. A version is always for a whole collection. Data objects and references inherit the version from their enclosing collection. Versioning is explicit, users have to create versions actively.

The following image displays a collection with references with no manually created versions.

After the creation of a new version, the data will look like this:

Semantic Annotations are copied when creating a version, just like the collection, data objects and references.

Permissions are collection-wide and across all versions.

When we create a new version of a referenced collection, the reference will move with the HEAD and the old collection will not be referenced anymore:

When we referenced an old version of a collection and a new version is created, the reference stays unchanged:

The generated schemas can be adapted using filters. For example, we use this to adapt the paths to match the root path of our api.

More information on this can be found here.

Quarkus sorts the list of path parameters in the OpenAPI spec alphabetically instead of by occurrence in the path.

For example, the following endpoint:

will lead to the following OpenAPI spec:

As we want the parameters to be sorted by occurrence in the path, this is not intended and can lead to issues in generated clients.

To fix this, we define the order of path and query params in the OpenAPI spec manually using @Parameter annotations. We do this for all path and query parameters.

For the above example, the result would look like this:

When using Java’s old date API, the generated openapi specification interprets the Date object as a date-only field.

The code snippet below:

generates the following openapi yaml snippet:

However, the Date object stores date and time, and should be treated by the openapi specification as an object that handles both date and time.

In the example, a modification on the createdAt field is needed to explicitly specify that the generated format should be in datetime format. The code snippet below adds a @Schema annotation, which specifies the format field:

This annotation then results in the following openapi specification containing a date-time format:

So, when using Java’s Date object of the old java.util.Date API, we need to explicitly specify the format in schema annotation, to generate a datetime object in the openapi specification.

To enable multipart file uploads in Swagger, the following schema is expected (Source):

Until now we weren’t able to reproduce this exact schema together with a proper file upload in the Swagger ui. Especially since we have requirements on the filename property to be not-null and required. So with annotations-only we could not reproduce this schema in Quarkus.

However, the following code snippet construct of interfaces/ classes and using the implementation field in the @Schema annotation, we were able to reproduce functionality for a file upload in the Swagger UI and a proper openapi schema.

This generates the following openapi specification:

This specification is rather complex and nested, but allows i.e., to add a required field to the UploadItemSchema schema, which is then generated as a required field in the Swagger UI.

One problem of this approach is that this construct of MultipartBodyFileUpload, UploadFormSchema and UploadItemSchema is needed for every REST endpoint that utilizes a multipart fileupload.

The solution is a combination of these two resources:

In order to support concurrent development of frontend and backend we decided to put the generated client for the frontend under version control (ADR-007 Client Generation for Frontend). The client can be found under backend-client. It’s exported members can be imported in frontend files like this:

We use junit5 for unit testing parts of our backend code. We aim to cover everything except endpoints by our unit tests.

To test the overall functionality of the backend we test our http endpoints with integration tests using @QuarkusIntegrationTest. In the pipeline, the tests are executed on a quarkus instance based on the build artifact built in the pipeline.

In order to perform endpoints tests we have implemented GenericEndpointsIT which will perform parametrized test on all the provided endpoints. This generic test will ensure the validity checks on parameters, perform access control checks, and expected status codes. Currently it is testing the below cases:

Test cases can be added to loadTestCases method by providing the inputs and expected outputs and codes. Every endpoint test case has the following properties:

For reference refer to the sample tests in the resource file generic_endpoints_test_cases.json.

We are using grafana k6 for load and performance tests. They are in the directory load-tests and are written in TypeScript. Tests can only be triggered on a local development computer but can be configured to use the local or the dev environment.

Global definitions like the font we use and typographies are defined as sass variables under 'nuxtend/styles/settings.scss'. Global overrides of component specific properties are also defined there.

The theme itself that mainly contains colors are defined in 'nuxtend/plugins/vuetify.ts'. The colors are taken over from the Style Guide that resides in Figma.

There are multiple possibilities to style vue components. We agreed on the following order when styling components.

In order to access the browsers local storage we make use of VueUse. It has a method called useStorage which gives us access to the local storage. With a key we can access the storage and fetch already stored data. If no data has been found it will fallback to the default value which can be provided in the function parameters.

To use session storage you can simply add "sessionStorage" as a third parameter.

Query and path parameters are validated using the Jakarta Constraints. These validations mostly rely on annotated constraints like: @NotNull or @PositiveOrZero in the backend’s REST classes.

This defines the dataObjectId to be not null and to be a positive or zero number. If these constraints are violated, a HTTP 400 Bad Request message is automatically returned. To make the @NotNull constraint work properly, it is important to use nullable types like Long instead of long with these validation annotations.

Currently, it is not possible to validate all Parameters in a meaningful way. For example, when a string is used instead of a numeric ID as a path parameter. This request: /collections/asdf, returns a 'HTTP 404 Not Found' error, instead of a 'HTTP 400 Bad Request' error. This behavior is intended by the framework itself and cannot be solved with annotations itself: https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest3x/jaxrs-resources.html#d0e2052. It is related to a type conversion error. There is also no way of running this validation check in the REST method code, since the HTTP 400 error is returned before the code in the REST method can be executed. There are two ways of fixing this:

The second approach is implemented for the version UUIDs. The version UUIDs are defined as a String in the REST method like: @org.hibernate.validator.constraints.UUID String versionUID. The @UUID annotation forces the UUID to be a string in the format of a valid UUID. If that is not the case, the API returns an 'HTTP 400 Bad Request'. In the REST method code, this UUID string is then converted to an actual UUID object:

Further resources: