shepard Architectural Documentation

Collections are meant as the root or container of your entire structure. All information, data, files, permissions, etc. are related to exactly one collection. Usually, a collection is the first thing you create.

In order to organize your project, you use Data Objects. They are very generic and can be used to organize your work as you wish. You can use them to create a tree based on experiments, lots, process steps or whatever you want.

There are different types of container available, e.g. for files, structured data, timeseries or semantic repositories. You can use them to group things together. If you want to store some images of a video camera, you can create a file container and upload them. If you need to store coordinates of a robot movement, you can create a timeseries containers and store the data there.

The Lab Journal enables researchers to document their experiments, data and results. Lab Journal entries can be linked to any Data Object. They also support html content, so that basic formatting and structuring can be used.

The backend is split into modules by their functionality. That means that there is e.g. a module for timeseries including managing containers and timeseries data, a module for managing files, for collections & data objects etc.

Each of those modules contains all technical components needed to fulfill it’s purpose, including endpoints, services, domain model objects, DAOs, entities and repositories.

Additionally, there may be modules for special functionalities like authentication.

We use quarkus-rest to define REST endpoints.

For authentication and general request validation, we use filters following the Jakarta REST way. The filters can be found at filters/. In general, all requests need to be done by authenticated users. The JWTFilter and the UserFilter take care of validating authentication.

Some endpoints should be public, for example /healthz and /versionz. To make this possible we use the PublicEndpointRegistry class. In this class we register all public endpoints in a static string array. The authentication filters will be bypassed for endpoints in this array. Since the /healthz endpoint is automatically public thanks to the SmallRye Health extension, we don’t need to add it to the PublicEndpointRegistry.

Each route is defined in pages/. The root file of a route itself should not contain a lot of logic, instead it should invoke one or more components.

In components/ all components are stored. These are grouped in folders by domain, e.g. components/collections.

Stateful logic can be extracted into composables and are stored in composables/.

Stateless utility functions should be stored under utils/.

For the interaction with the backend we check a generated openapi client into the repository, see here for more information.

In order to directly instantiate the API clients with our default configuration, we use the createApiInstance utility.

The current LTS version of Node.js should be used in all of the project (Gitlab jobs, docker images, local development, etc.) to maintain reliability and performance.

The current LTS version of Quarkus should be used in our backend to maintain reliability and performance.

The following artifacts are provided as documentation of shepard:

Unresolved directive in src/08_concepts/index.adoc - include::migrations.adoc[]

To be able to authenticate in the frontend and acquire a JWT token, we use @sidebase/nuxt-auth as our authentication module in the frontend.

We handle the merge requests opened by renovate by performing the following steps for each update:

Also, the dependencies in package-lock.json should be updated. This is done by running npm update in the top level directory.

In case we could not perform the update it should be suspended and documented in the list of suspended updates. The reason can either be too much effort (we create a new story for that update) or that the update is not possible or feasible right now.

This can be done by excluding the library or specific version in the renovate config. Afterwards the config needs to be merged to main with the following commands:

Afterwards the merge request can be closed.

Sometimes, when the configuration changes or dependency updates were done without renovate, the bot might abandon a merge request. In this case the merge request is not automatically closed and has to be closed manually. The corresponding branch must also be deleted manually to keep things clean.

BARTOC knows about terminology registries, including itself. Registries also provide access to full terminologies either via an API (terminology service) or by other means (terminology repository).

https://bartoc.org/registries

Typical "interfaces":

(others could in include eclass or IEEE iris)

Query documents using native mongoDB mechanics

Query collections, data objects and references

Usually a new shepard version is released on the first Monday of the month. However, this date is not fixed and can be postponed by a few days if necessary. This monthly release increases the release version number.

We use semantic versioning, meaning that the version number consists of a major, minor and patch number in the following format: MAJOR.MINOR.PATCH. Minor is the default version increase for a release. Breaking changes imply a Major release. And hotfixes or patches are performed as a Patch release (see Hotfix process below).

Currently there are two workflows for releases, one minor/ major release and a patch release. Both release types are explained step by step below.

These steps describe a regular (monthly) release for shepard but can also be used to release an unplanned patch release.

Furthermore, there are two ways to create an unplanned patch/ hotfix release.

The first option is the more classical hotfix approach, meaning that it only brings the changes from merge requests containing hotfixes from the develop branch to the main branch. The steps needed for this option are explained below in the section: Performing a hotfix release.

The second option includes creating an MR containing the needed patch changes on develop branch, then merge the develop branch on main and create a new minor release. This would create a new out-of-cycle release containing the patch and all changes from develop since the last release. This option is the same procedure as a regular release, which is described right below.

Hotfixes are changes to the main branch outside the regular releases to fix urgent bugs or make small changes that need to be applied to the default branch of the project. The steps below describe how one can release a single hotfix MR without having to merge the develop branch on main. This means that the other changes on develop branch are only merged when a new regular release is created.

This section is a short summary of this page.

This section is a short summary of this page.

As a shepard user I want to be able to use different versions of data sets to facilitate collaboration in a research project and lay the groundwork for future features like branching, visualization of differences or restore functionality.

I can define a version of a collections to mark a milestone in the project in order to freeze the status that the data set has right now via the API. There is always the one active version called HEAD which is the working copy that can be edited on shepard and there can be n versions on shepard that are then read-only. If I never define a version as a user then there is no change in functionality for me. Versions are identified by a UUID. Version should be applied for organizational elements, not for payload data. A version is always for a whole collection. Data objects and references inherit the version from their enclosing collection. Versioning is explicit, users have to create versions actively.

The following image displays a collection with references with no manually created versions.

After the creation of a new version, the data will look like this:

Semantic Annotations are copied when creating a version, just like the collection, data objects and references.

Permissions are collection-wide and across all versions.

When we create a new version of a referenced collection, the reference will move with the HEAD and the old collection will not be referenced anymore:

When we referenced an old version of a collection and a new version is created, the reference stays unchanged:

The generated schemas can be adapted using filters. For example, we use this to adapt the paths to match the root path of our api.

More information on this can be found here.

Quarkus sorts the list of path parameters in the OpenAPI spec alphabetically instead of by occurrence in the path.

For example, the following endpoint:

will lead to the following OpenAPI spec:

As we want the parameters to be sorted by occurrence in the path, this is not intended and can lead to issues in generated clients.

To fix this, we define the order of path and query params in the OpenAPI spec manually using @Parameter annotations. We do this for all path and query parameters.

For the above example, the result would look like this:

When using Java’s old date API, the generated openapi specification interprets the Date object as a date-only field.

The code snippet below:

generates the following openapi yaml snippet:

However, the Date object stores date and time, and should be treated by the openapi specification as an object that handles both date and time.

In the example, a modification on the createdAt field is needed to explicitly specify that the generated format should be in datetime format. The code snippet below adds a @Schema annotation, which specifies the format field:

This annotation then results in the following openapi specification containing a date-time format:

So, when using Java’s Date object of the old java.util.Date API, we need to explicitly specify the format in schema annotation, to generate a datetime object in the openapi specification.

To enable multipart file uploads in Swagger, the following schema is expected (Source):

Until now we weren’t able to reproduce this exact schema together with a proper file upload in the Swagger ui. Especially since we have requirements on the filename property to be not-null and required. So with annotations-only we could not reproduce this schema in Quarkus.

However, the following code snippet construct of interfaces/ classes and using the implementation field in the @Schema annotation, we were able to reproduce functionality for a file upload in the Swagger UI and a proper openapi schema.

This generates the following openapi specification:

This specification is rather complex and nested, but allows i.e., to add a required field to the UploadItemSchema schema, which is then generated as a required field in the Swagger UI.

One problem of this approach is that this construct of MultipartBodyFileUpload, UploadFormSchema and UploadItemSchema is needed for every REST endpoint that utilizes a multipart fileupload.

The solution is a combination of these two resources:

In order to support concurrent development of frontend and backend we decided to put the generated client for the frontend under version control (ADR-007 Client Generation for Frontend). The client can be found under backend-client. It’s exported members can be imported in frontend files like this:

We use junit5 for unit testing parts of our backend code. We aim to cover everything except endpoints by our unit tests.

To test the overall functionality of the backend we test our http endpoints with integration tests using @QuarkusIntegrationTest. In the pipeline, the tests are executed on a quarkus instance based on the build artifact built in the pipeline.

We are using grafana k6 for load and performance tests. They are in the directory load-tests and are written in TypeScript. Tests can only be triggered on a local development computer but can be configured to use the local or the dev environment.

Global definitions like the font we use and typographies are defined as sass variables under 'nuxtend/styles/settings.scss'. Global overrides of component specific properties are also defined there.

The theme itself that mainly contains colors are defined in 'nuxtend/plugins/vuetify.ts'. The colors are taken over from the Style Guide that resides in Figma.

There are multiple possibilities to style vue components. We agreed on the following order when styling components.

In order to access the browsers local storage we make use of VueUse. It has a method called useStorage which gives us access to the local storage. With a key we can access the storage and fetch already stored data. If no data has been found it will fallback to the default value which can be provided in the function parameters.

To use session storage you can simply add "sessionStorage" as a third parameter.

Further resources:

The following process handles all technical debt except for dependency updates, which have their own process. Besides dependency updates, every other technical debt is to be documented in the form of an issue in the backlog.

If a technical debt has been identified and a corresponding issue has been created in the backlog, the usual planning and prioritization process takes care of this debt. This makes the backlog the single source of truth for all known and unresolved technical debt.

Usually, the technical debt can be resolved in this way. In rare cases, it can happen that we want to keep this debt or decide that the debt is not really a problem for us. In these cases, the situation needs to be described in the table below and the corresponding issue can then be closed.