[hma] Clean up documentation story

[cassidyjames]

We've heard from HMA adopters that the current state of HMA documentation is a bit messy to follow. Right now there are a bunch of places to look for information:

• README.md in the HMA subfolder
• /docs in the HMA subfolder
• This repo's wiki
• ThreatExchange GitHub Pages
• ThreatExchange docs on Meta for Developers
• API documentation within the web app ([hma] Add OpenAPI docs using flask-openapi3 (#1825) #1891)

It could help adoption and adopter productivity to clean this up a bit. :) My proposal, if possible, would be to:

• Consolidate HMA technical documentation either under /docs or on the wiki
  • Update the README to be a bit more concise, moving its technical documentation over to the rest of the docs
  • If under /docs, optionally spin up a static docs site to make that a bit more browsable
• Retire the old GitHub Pages site
• Link to the canonical HMA docs location from the ThreatExchange docs
• Host the generated API documentation… somewhere, and cross-link between it and the new canonical docs location

I'm almost certainly missing some context, but I'd love to hear from other contributors and adopters if this (or an even better!) plan would make sense and is something we can work towards.

[Dcallies]

README

Every github project should have a readme, which has a brief introduction to the product, and a link to where the other docs are. Folks in open source are trained to check it first.

Wiki

The wiki was primarily to document HMA 1.0, as is just a skeleton at this point, we can delete it

Github Pages

I don't even remember this being made, but it looks like it's only a skeleton, unless someone in the community wants to volunteer to revive it, we maybe should just delete it

/docs directory

This was created directly as the result of the previous community discussion for #1825, where folks seemed to prefer topical .md files

ThreatExchange API Documentation

The ThreatExchange API docs describe another product - ThreatExchange (as described by it's link). It's unfortunate that this github repository shares the same name as the ThreatExchange platform, but they are different things. The ThreatExchange API docs are roughly equivalent to the NCMEC hash API docs at https://report.cybertip.org/ws-hashsharing/v2/documentation/. I don't think NCMEC would say it would be correct to point to the HMA docs from that site, so I don't think it makes sense to point to HMA docs from the ThreatExchange API either.

In-app docs

These are directly computed from the true state of the API, so would argue we should keep these. They are useful because they represent the real API for the deployed version of the code. There is almost certainly a way to export this from the code and re-host it elsewhere.

David's take:

The problem isn't necessarily that there are multiple places with information, but that there's no one place to peruse all of it. Most projects I've seen have documentation on a website or pages product, and not in the code. However, I don't know how to host and maintain those types of pages at the moment. If folks have the time, I'd say:

1. Take a look at how other projects are hosting their own documentation pages like
   1. Black's read the docs: https://black.readthedocs.io/en/stable/
   2. Flask's sphinx: https://flask.palletsprojects.com/en/stable/
   3. Something else that seems standard
2. Identify one that can pull the resources we want together in one place
3. Build up that new solution
4. Delete all the other copies that can't be pulled into the new solution
   1. As long as the other locations are automatically pulled into the new solution, we don't have to delete them. E.g. the self-hosted docs are "duplicate", but as long as they automatically sync to the central source of truth, there's no cost to having the second copy

[Dcallies]

I've removed both the pages and wiki to reduce the number of surfaces.

[ThisIsMissEm]

/docs directory

Yeah, if my memory serves, this was largely just to get us something as opposed to nothing.

so I don't think it makes sense to point to HMA docs from the ThreatExchange API either.

Agreed.

---

On tooling, I'd strongly recommend Vitepress, as it's just markdown but rather easy to configure and theme the built documentation site output. We can also support having documentation for each part of this repository that way too. (Example: FediMod FIRES site and the source)

As it's just markdown, pages can also be automatically generated in a build step if necessary, and there's an OpenAPI plugin available too.

[cassidyjames]

@ThisIsMissEm that sounds great to me! I feel like we could probably bikeshed which docs generator to use forever, but those Vitepress docs look really nice and the straight-up Markdown source makes me happy. 😁

Another option that was pointed out and looked nice to me at a glance was mdBook, but as long as it's straightforward, can be automated in CI, and gives us a static site that looks good I'm not that picky personally!

[ThisIsMissEm]

@cassidyjames I can try to find some time to start putting together a branch for this, but I'm a bit busy this and next week with other work. Could we decide if the documentation site will cover just HMA or if it'll cover the entire repository, since we only get one GitHub Pages website?

I think the entire repository is probably the way to go, but I'm not certain on that.

[ThisIsMissEm]

@Dcallies what license is the contents of this PDF document under? I'm assuming BSD, but just want to check, because it's content that we could definitely include on a documentation website: https://github.com/facebook/ThreatExchange/blob/main/hashing/hashing.pdf

[Dcallies]

Everything is BSD unless otherwise noted at the beginning of the file or in the LICENSE file of the local directory.