JS Menu Component Initiative Questions

This document is intended to capture some initial general architecture thoughts/questions/concerns to help define scope

Mission Statement

Provide the best way for JavaScript front-ends to consume configurable menus managed in Drupal

Success is

  • An official React component for menus
  • An official Vue component for menus[a][b]
  • Has a better developer experience than Contentful
  • Maintain the current Twig templates for traditional sites

Parts to figure out and create[c][d][e]

  • Menu web service API
  • Module-defined menu items[f]
  • User-defined menu items
  • Menu hierarchies
  • Path aliases
  • Routing
  • Testing
  • Documentation
  • npm package
  • Releasing to npm
  • Security management
  • Managing dependencies

Architecture Questions and Concerns

We’re not necessarily looking for specific answers here, this is primarily to figure out different areas we’ll need to work on

Routing

  • Sally: Do we first need to create a router? Menu links might not be very useful if they point to places the application can’t understand or do anything with. Some links are going to be internal to Drupal so won’t be useful (e.g. the login / logout links).

Sally: Will the routing system be separate to Drupal? The set of paths used in an applications applications won’t necessarily be the same as what’s used within Drupal. Perhaps it has its own /user path. See https://www.drupal.org/project/snail

  • Gabe: I don’t think we need to create a router. If we make path aliases work seamlessly with JSON:API, I would hope that there would not need to be a difference in routes. Decoupled Pages was my attempt to bridge the gap between FE/BE routes by loading SPAs from Drupal managed routes.[g][h][i]
  • John: Lots of React code bases already have routers, e.g. Gatsby. If we write loosely coupled components, we can just use a pre-made router. [j]
  • Gabe: I would like for us to explore/promote a different pattern for routing when building JS SPAs powered by Drupal. JS-managed routing disempowers site builders and content editors. E.g. it becomes impossible to create a new content type with a new pathauto pattern without involving a developer.[k]

URL aliases

  • Gabe: How do we handle URL aliases? Should the menu JSON objects point to /jsonapi/** endpoints or should they point to the URL aliases?

Multilingual

  • Sally: Will there be options for different ways to handle multilingual? e.g. showing an untranslated item if no translation is available vs showing nothing, or just provide enough information that the consumer can filter them and decide. Will we provide recommendations for building multilingual menus? (Some sites have a separate menu for each language to work around that quirk)
  • John: I think this is a good justification for having a separate component for the data queries. JSON:API needs a separate query for different languages. If we have one component that queries a specific language[l], we can just pass the data to the menu renderer.[m][n]

Release, security management

  • Sally: How do we manage releases and security updates for JavaScript dependencies since they could be on a different cadence (assuming we use https://github.com/dependabot to manage getting upstream updates)[o]
  • Sally: Do we work on GitHub (where the rest of the JS community is)?[p]
  • Gabe: I would love to work on GitHub. But I feel like it’s an obstacle we’d be adding by choice.  I.e. it’s going to add an additional step to each element of “how do we ship JS components?” (e.g. does this have security team coverage? How do you report vulnerabilities?)
  • Sally: I think you’d have a lot less obstacles shipping JS components from GitHub than drupal.org - it’s easier to set up integration with npm and you get automated dependency updates[q][r][s][t][u][v].
  • Théo: Yes to all in github. We tried the “have the js devs come in Drupal” and it doesn’t work. Let’s try the other way it should be more successful. [w][x]

Testing

Liam: Would it be prudent to begin this project with testing in mind? Something like Jest, as it is flexible and able to test either Vue, or React.

  • Edgar: I would suggest cypress for testing[y][z][aa][ab][ac][ad]
  • Larowlan: I would recommend the @testing-library suite of testing tools that provide utilities on top of jest for best-practices testing dom / react / vue[ae][af]

Authentication

  • Sally: How do we handle menu items for different roles / logged in users or do we only handle items for anonymous users to start with?[ag][ah][ai]
  • Gabe: I think we can/should handle all cases. I think it’s really a matter of the API using appropriate cache contexts etc. I.e. if the component merely presents what is given to it, the back end can be responsible for hiding things.
  • Sally: Are we only going to tackle displaying menus i.e. not creating and updating[aj] them
  • Gabe: I hope that we begin with the read-only use case. That is smaller and scope and will probably illicit more contribution.[ak][al][am]

Miscellaneous

  • Sally: How do you handle links like Login / Logout[an]?
  • Gabe: Can we let them go to Drupal? I think most people should deploy their SPAs to the same domain as the Drupal site and can use cookie auth.[ao][ap] If they  take over `/user/login` with an SPA and have the SPA served from that URL[aq][ar][as][at][au] it could use the REST login endpoints to get a `Set-Cookie` header.
  • John: We just display them like normal. It’s up to the React app on how to handle a path to `/user/login`.
  • Gabe: *runs and hides*
  • Sally: How do you efficiently fetch a menu i.e. avoid having to first get a list of menus to filter out the UUID of the one you want, extra requests to fetch the links etc
  • Gabe: with a new JSON:API endpoint of course! :P I.e. let’s make a new resource type called `menu` that abstracts away the underlying content/config entities.[az][ba][bb]
  • Josh: Who will create the issue?
  • Josh: Can we collaborate on the issue write up before it’s created?[be][bf][bg]
  • John: Do we want to create a group on https://contrib2020.getopensocial.net/all-groups ?
  • Mahmoud:
    I think we can start with the     admin panel since it has the base blocks for the CMS by:
    1- Build the infra for the theme “like Claro theme” so it can understand the “Meta data” API’s that define the component itself.
    2- Create the documentation for the new architecture so the modules & themes maintainers can start preparing their projects to follow the new API’s for both Backend and Frontend.  
    3- Start converting the main blocks to React components to work with the new component-based theme:    [bh][bi]
  • Page title block.
  • Primary tabs block.
  • Secondary tabs block.
  • Breadcrumbs.
  • Status messages.
  • Help.
  • Primary admin actions.
  • Main page content[bj][bk][bl][bm][bn][bo].

So when we deliver this in the Decoupled release "Experimental" we will have the following benefits:

  • The community will have the time to test and enhance the new structure.
  • The community will follow the same approach to build the reset of components.
  • When we have something stable in structure we can invest in automating tools which can help to migrate projects to the decoupled version “Decouple Bot”.

Next steps

  • Define at a high level, how do we plan to solve this
  • Document the process and where are we planning to work
  • Where will the work be done[bp][bq] (Drupal.org vs Github)? Will something be added to core as a new experimental module https://www.drupal.org/core/experimental, or do we need to make changes across existing subsystems and therefore need a feature branch?
  • Define a rough roadmap (what and when) and proposal on how it should be tied to Drupal Core release cycle.

Discussion on 28th of July 2020

Q: Where will the work be done (Drupal.org vs Github)?

A: Github seems like the most practical option because it has a lot of the tooling needed pre-built. We will create a list of missing features in Drupal.org so that we can estimate what it would take to make Drupal.org support JavaScript projects the same way Github does. We could also look into the JavaScript tooling of Gitlab to see if it could be used instead of Github. Drupal.org is currently utilizing Gitlab in the backed, so it might be easier to use it as a platform rather than Github. However, the follow-up questions are:

  • How are we going to version the library? When will the releases be made?
  • Are there special considerations due to the speed of JavaScript libraries?
  • How do we get security team coverage for the library?

Q: Will something be added to core as a new experimental module https://www.drupal.org/core/experimental, or do we need to make changes across existing subsystems and therefore need a feature branch?

A: At this point, we are planning to only make improvements to JSON:API by filing issues against the core module. We can re-evaluate this if it turns out being something that would be required. If we want to accelerate the development of the JavaScript libraries, we can create a contrib project that applies a collection of patches on core.

Q: What is the outcome of the initiative?

A: Documentation on how to setup a React or a Vue application from scratch that can fetch menu links from Drupal. This documentation will be extended to include other examples.

Discussion on 28th of August 2020

  • What does it mean to have a JavaScript menu component. Is it supposed to be something that can be used and extended as-is (and maintained accordingly), or more of a demonstration to base a documentation around?
  • What's "success" for the initiative?
  • What's the relationship with existing core code?
  • How many people should maintain the newly added code? What is the relation to the existing maintainers?

Prior Art

Dump all your links here!

  • Benjy: Algolia provides some of the best React components I've worked with, very robust, easy to extend and they build on-top of their underlying JavaScript libraries that handle much of the communication with the server. Could be a good place to look for reference architectures.[br]
    I specifically like how each component exposes a “connect” method that lets you implement your own rendering for the entire component but with all the data/handlers passed in as props.
  • Liam: Other popular component libraries worth looking at for reference

John Albin: We’ve built and iterated on a React component that handles Drupal menu data for a few sites. This is the criteria we used for what the code is responsible for:

  • authentication and authorization are handled by some other code that fetches the data and hands it to the React component to render. (In our case it was a GraphQL query using a Drupal regular user authentication cookie.) This point means the React component for menus did NOT do data fetching.
  • Internationalization is handled by the data fetching code too, but the component uses react-i18next to translate any additional user text the implementation needs.
  • It needed to render the menu data using radically different HTML and CSS. For example, the same React menu component was used to render the main menu on mobile (with opening/closing buttons) as was used to render the footer menu (simple links). This means it had to be highly configurable: the link item, the link, the link text were all separately configurable about how to render those parts and with what HTML.
  • It had to handle nested menus just like the up-to-8-layers-deep menu system in Drupal.
  • It had to handle all the data in a Drupal core menu link: url, link text, description.
  • It should pass any extra data added by contrib to its rendering parts.
  • By default, it renders the menu data using nested unordered lists. More advanced styling requires configuring the bits you want to override.
  • It has a “show children” option (that is off by default) that uses React state to show/hide child links.
  • The depth of any menu link is tracked and passed to each menu link so it can change the rendering based on its depth.
  • TypeScript![bs]

[a]Does react & vue have an official status as frontend framework or is it just based on expressed motivation ? (bottom line : I could provide resources to build an Angular component, even a generic WebComponent build with Angular, but I won't invest time if only react & vue are allowed atm)

[b]I understood it as saying react AND ...other frameworks, not limited to vue. It seems like there could be some middleware that handles the API responses and parses it into something more easily usable, and we'd create example components in _at least_ react and vue to show how it would work.

[c]todo: what blocks what -> know where to start.& identify what can be done in parallel

[d]Is this list complete-ish?

[e]This list could be a TOC with a section for each item, outlining what's already know & what needs to be learned next

[f]Given https://www.drupal.org/node/3158562 - request to consistently refer to these as "Menu links" everywhere. ;)  Thanks!

[g]We built something similar for the Admin UI -

https://github.com/jsdrupal/drupal-admin-ui/tree/master/packages/extension-points#routes-configuration

and

https://github.com/jsdrupal/admin_ui/blob/8.x-1.x/modules/admin_ui_widget_example/admin_ui_widget_example.admin_ui.routes.yml

fwiw I'm against the approach of mixing the routing systems. What I usually do is use React Router with a wildcard as the final fallback route, which will look the path up in Drupal (with Snail enabled). Then you can define whatever routes you want in your FE application and they don't interfere with each other

[h]Instead of using the URL to pick components (i.e. what a FE router does) before navigation, have you API responses to pick components? E.g. if you get an API response with `type: node--article` the app picks a component for that response instead of the right component for the URL?

I'm thinking about "Fetching before navigation": https://router.vuejs.org/guide/advanced/data-fetching.html#fetching-after-navigation

I know that this isn't a super conventional approach, but I think there are some real advantages in promoting this sort of approach for browser-based things.

[i]We did something similar last year where we had wildcard routes in the frontend router and the response included a `type` property. The type property was the name of the component to load lazily. This helped content editors to add/remove pages without the need of developers.

This of course required changes to the API response.

[j]agree

[k]+1, path (& aliases) should be considered part of the content (the node) and just be a blob of text for the frontend (which then rely on a resolution service that must be provided by Drupal)

[l]We shouldn't be hesitant to create endpoints that don't already exist on the Drupal side. I.e. let's not work around JSON:API, but make JSON:API more flexible. The more business logic that we can push to the server, the better. E.g. we can/should make JSON:API respect the Accept-Language header for multilingual menus if that means writing less JavaScript.

[m]agree

[n]+1 to fetching data outside of the display component.

[o]happy to be a bridge to the security team. orgs like Symfony have a private repo on github for security work

[p]I think working on github should be our preferred option, the drupal project tooling is built around php projects

[q]This is a really good reason to go this direction.

[r]Maybe we can set something up that mirrors the d.o repository to GitHub or vice versa?

My fear is that if we go off to Github, core committers won't give us the credibility/attention we need (for better or worse). Maybe that's an unjustified fear?

[s]I can not speak for core contributor support, but what sort of support would we require?

I would argue that Github could extend the exposure of Drupal, and perhaps bring in new contributors who otherwise wouldn't be involved in the project?

[t]I know the feeling, just to keep a frame of reference, If we attract like 2 people as long term contributors I would say it's a success.

[u]On Gabriel's concern, I was thinking a bit about issues in core that will inevitably need to be resolved. But I wouldn't imagine core contributors intentionally not giving attention to  an issue that is  part  of an initiative. Again, though, I am very new to the community here... so my knowledge is very weak on that front.

[v]Even though this isn't the only reason to go with Github, however, we can setup Renovate or Dependabot on Gitlab as well.

[w]Now that d.o. is gitlab-based, couldn’t it be  "all on gitlab" instead of "all on github", with an eye to making this somehow bridged with d.o. .

[x]Could be the safe choice, Another valid question though would be how people coontribute. Would we expect to grow contributions to outside the Drupal community. If so, the contribution flow on drupal.org/drupal gitlab is not very consistent at the moment with how the community outside drupal works.

[y]What benefits does Cypress bring vs Jest?

[z]- Very easy to get started and use

- Nice visual testing

- Easy to spot issues while running a test and debug

[aa]This just my opinion

[ab]Okay, so I guess Cypress is more for the acceptance testing, where as Jest is more focused on Unit testing.

My original suggestion was in terms of the code side, to ensure refactors and contributions don't break the functionality. This could also be integrated with Github actions to ensure bad commits are avoided (if this project ends up going that direction)

The visual end of a component should be flexible, and so I question the value of a visual testing framework at the library level? I would think that something like Cypress would be contextual to the theme/project that is being built using the component library, no?

[ac]Does Cypress allow you to run offline without a dependency on their paid service?

[ad]This conversation has some great info, thanks! I'd like to keep this space for higher level scope and architecture so I don't think we're ready to dive too deeply into this yet (other than to say we should definitely have tests 😉)

[ae]cypress and jest serve different needs and and different speed scales. I do most of my Vue tests in Jest, and only use Cypress on end-projects because it is so slow and applying to integration issues.

[af]We use the same here for our VueJS projects.

[ag]So perhaps having an AuthProvider context would also be a hard-requirement here. i.e. something drop-in like the auth0 things.

[ah]Can you expand on this? I'm not following your mental hops.

[ai]something like https://github.com/auth0/auth0.js - there was a gatsby days presentation on the topic https://speakerdeck.com/samjulien/a-recipe-to-power-up-gatsby-with-auth

[aj]Immediately thought of tabledrag when I saw the slide. So it'll stay around a little bit longer :p

[ak]agree

[al]yes. Focus on what's needed to be able to show a "click here" thingy :)

[am]+1

[an]What different types of links are there to consider?

[ao]I don't think that's very practical - though I agree that cookie auth is so much easier. Maybe we can compromise and stipulate they should at least be on the same root domain?

[ap]sure :O)

[aq]This won't work for apps e.g. React Native

[ar]oooh, good point.

[as]Maybe that's only a hypothetical problem though. Would you really put a login/logout link into a menu used by a native app?

[at]sure, why not! 😂

The answer here could be that we don't support any "special routes", only node paths or custom routes which are specifically marked as content somehow

[au]Although one of Dries' goals was to have this as a component which can help build the Admin UI, so I guess we'll have to tackle it one way or another

[av]these modules should be integrating with the serializers - we can't special case them

[aw]having a dedicated data component would allow some form of handling for this right?

[ax]Ah, I didn't mean specifically how the data is transformed, I meant that the API schema could shift suddenly with field changes

[ay]This is a very hard problem afaik. The question might also be, for at least the menu data, has that happened before? How many time? I do know that in some ecosystems you will have specific major versions of packages support specific versions of the framework it uses.

For content itself, Drupal doesn't really support versioning of jsonapi at least.

[az]🎉

[ba]+1

[bb]Hehe, easy! And just relate to itself, menu-ception!

[bc]feels like that's not part of the 80%

[bd]Agree

[be]I suggest @lauri.eskola@acquia.com as he's the front-end framework manager

_Assigned to lauri.eskola@acquia.com_

[bf]Make the issue write up part of this doc?

[bg]I definitely think we should :)

Maybe +lauri.eskola@acquia.com can create the issue with a blank issue summary and link to this doc so that this doc is more discoverable.

_Assigned to lauri.eskola@acquia.com_

[bh]the reason for starting small (just a menu component) is because this is too ambitious.

[bi]From past experience - yes very much so 😂😭

Also bearing in mind subsystems in Drupal are actively being worked on, so then you have multiple moving targets to keep up with

[bj]Should this be a component, or something more granular, like a collection of Cards, Media Items, etc?

[bk]Perhaps even a "container" component, which would be a parent to much smaller pieces?

[bl]I think it can be a container of any other components, like the layout builder Approach.

[bm]this is out of scope here, we're trying to build a single well defined component as a proof of concept. Attempting to rebuild everything in one go is a bridge too far

[bn]I understand from what Dries mentioned that  is the menu component is just  ofan example of the big idea.

Focussing on one component is good thing, hover I think it's

better to test how these components and its infrastructure work together.

[bo]I think we probably do want to avoid scope creep. It's okay to think about the implications, and to think about how future components will be introduced later on. But I would tend to agree with Lee, that rebuilding everything is too much for phase 1.

[bp]More importantly we should agree on how are users going to use/test this component or feature. a dedicated module/distribution/theme? What is knowledge and effort expected for people who want to try this out

[bq]+1

[br]oooh, this is a great recommendation :) Thank you!

[bs]This is probably a really good call.