Installing Clowder

What type of user are you?

For most users of Clowder: Get started here 👇

For developers of Clowder itself: Dev quickstart here 👇

  • Build Clowder from source via IntelliJ’s Play-2 run configuration.

  • Run the required services via Docker.

For production instances of Clowder, a Kubernetes deployment is recommended and manual installations are being phased out.

Users of Clowder: Getting Started via Docker

  1. Install Docker Desktop (if you haven’t already)

  2. Clone or download Clowder on Github (use the default develop branch)

git clone https://github.com/clowder-framework/clowder.git
  1. Navigate to Clowder’s root directory in your bash command line (cd clowder)

  2. Start Clowder using the Docker Compose configuration, via your command line

docker-compose up -d
  1. If you experience any issue with file uploads and see the below error message in the console:

[ERROR  ] - application - Could not create folder on disk /home/clowder/data/uploads/xx/xx/xx
[ERROR  ] - application - Could not save bytes, deleting file xxx

you can try this command:

docker-compose exec -u 0 clowder chmod 777 /home/clowder/data
  1. Open your web browser to localhost:8000

  • If you see Error 404, allow a minute for it to appear.

  • Note: use port 8000 for docker run. However, port 9000 for manual builds.

⭐ If you experience any trouble, come ask us on Slack here! ⭐

Helpful docker commands
  • docker-compose up -d - start up all required services

  • docker-compose down - stop all docker containers

  • docker-compose logs -f - see the logs

  • docker ps - check how many services are running

  • docker info - details about your docker version

    • After starting Docker, check that your services are running via the Docker Desktop GUI, or run docker ps and check that 3 containers are running.

    • The “image” column should show rabbitmq, elasticsearch and mongo.

Clowder started! Now create a new user 👇

Sign up for a Clowder login account

After installing Clowder, you still need to sign up for a user account.

Run this in your terminal to create a new account:

docker run --rm -ti --network clowder_clowder -e \
   FIRST_NAME=Admin -e LAST_NAME=User \
   -e EMAIL_ADDRESS=admin@example.com -e PASSWORD=catsarecute \
   -e ADMIN=true clowder/mongo-init

Optionally, edit these properties to your liking:

  • FIRST_NAME

  • LAST_NAME

  • EMAIL_ADDRESS

  • PASSWORD

  • ADMIN (only set this if you want the user to have superadmin rights, make sure at lease one user has this).

✅ Configuration complete! Now you can login to Clowder via localhost:9000 in your browser.

Warning

If you renamed the base clowder folder to something else, like kitten, then the --netowrk parameter must be changed to --network kitten_clowder.

All done! You should be able to login to your new account, create new Spaces & Datasets and upload many different types of data.

Note

Before you go, check out useful information like the Clowder ‘All Paws’ YouTube playist.

Try the default extractors for simple quality of life improvements in Clowder.

$ docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.extractors.yml up -d

Clowder Developers: Getting Started

For Clowder developers, a hybrid is recommended:

  • Part 1: Run the required services via Docker, and expose each of their ports to Clowder.

  • Part 2: Run the Clowder instance manually via IntelliJ Ultimate’s Play-2 run configuration.

Part 1: Setup Docker

  1. Install Docker (if you haven’t already)

  2. Clone Clowder’s develop branch (the default)

git clone https://github.com/clowder-framework/clowder.git
Apple Silicon M1 users, additional instructions here 💻👈

Clowder works well on Apple Silicon, with only one minor caveat. No changes are necessary, but these optimizations are handy.

Elasticsearch does not work and so the search bar in the top right of the web interface will not work or be visible. Clowder depends on an older version of Elasticsearch before it added Apple Silicon support, and Docker’s QEMU emulation of x64 happens to fail causing the container to infinitely crash and restart.

To prevent this container from constantly crashing and restarting, please comment it out of the Docker definition in docker-compose.yml.

# COMMENT THIS OUT in docker-compose.yml:

# search index (optional, needed for search and sorting future)
elasticsearch:
  image: clowder/elasticsearch:${CLOWDER_VERSION:-latest}
  command: elasticsearch -Des.cluster.name="clowder"
  networks:
    - clowder
  restart: unless-stopped
  environment:
    - cluster.name=clowder
  volumes:
    - elasticsearch:/usr/share/elasticsearch/data

Additionally, you may have to install Scala and SBT on your Mac.

brew install scala sbt

Finally, there is no need to specify a ‘default Docker platform, and could hurt performance. (i.e. <DO NOT> export DOCKER_DEFAULT_PLATFORM=linux/amd64.’) Only the necessary Docker containers will automatically emulate x64, and the rest will run natively on Apple Silicon.

Expose Docker services’ ports to Clowder

In order for Clowder to access the required services (at a minimum: MongoDB, ElasticSearch, RabbitMQ. See Requirements for details.), we must tell Clowder which ports the services are using.

  1. Create an override file, where we will store the port information

# navigate to Clowder base directory
cd clowder
# create new file docker-compose.override.yml
touch docker-compose.override.yml
  1. Copy and paste the lines below into that file we just created docker-compose.override.yml

  1. Test that our services work! First start them:

docker-compose up -d

Note

By default, running docker-compose up -d uses the docker-compose.yml configuration and will apply overrides found in docker-compose.override.yml. Neither file need to be specified on the command line.

(Optional) Check that the Docker containers are running

You can see them in the Docker Desktop application, or in the web browser shown below.

localhost:27017 - You should see: “It looks like you’re trying to access MongoDB…” Success! localhost:15672 - You should see: the RabbitMQ login screen (no need to login tho!). Success!

Now keep everything running, and next let’s build Clowder from source 👇

Part 2: Run Clowder via IntelliJ

  1. Install IntelliJ Ultimate Edition.

    • This guide will assume developers use IntelliJ. Ultimate Edition is required for the Play2 configuration.

  2. Open the base Clowder directory & install Scala plugin

    • This should prompt you to install the Scala plugin! Install it.

    • Or, manually install the Scala Plugin for IntelliJ File –> Settings –> Plugins –> Download Scala.

  3. Install Java 8 (i.e. Java 1.8) on your computer. Clowder requires Java version 8 and is not compatible with other versions.

    • I find this easiest to do via IntelliJ’s Plugin Manager. File –> Project Structure –> SDKs –> + icon –> Download JDK

    • Select Version 1.8 (Clowder is only compatible with Java 8 (1.8), nothing higher) –> Vendor: Eclipse Temurin (AdoptOpenJDK Hotspot) –> Download.

    • Alternatively, download the JDK online at AdoptOpenJDK. Java 8, HotSpot.

    • Then point IntelliJ to the JDK folder under Project Structure –> SDKs and specify the root folder of the JDK you just downloaded.

Download JDK from IntelliJ.
  1. Add a new Run Configuration

    • In the top right, click the dropdown and click “Edit Configurations…”

Add new configuration
  1. Create a new Play 2 App configuration

Note

If you don’t see Play 2 App in the list, ensure you have the Scala plugin installed. If Play2 still isn’t there, you may need to use IntelliJ Ultimate version (instead of Community). I experienced this bug, feel free to ask in the Clowder Slack here.

Create play2 configuration.
  1. The default run configuration should be okay, see image below.

Specify the JDK path in the Run Configuration.

The default Clowder run Configuration.

Note

Later, if Clowder feels slow (multiple seconds per page load) then you will need to add JNotify to your JVM Options on this page. See the instructions at bottom of this page.

⭐️ Now start Clowder: In IntelliJ, click the green play button ▶️ (top right) to build Clowder from source! Give it a minute to finish. Access Clowder via localhost:9000 in the browser.

Also note, a handy debugging mode is enabled by default. You can run the debug mode by clicking the green “bug” 🐞 button right beside the play button.

Creating a local Clowder account

After installing Clowder, you still need to sign up for a user account.

Run this in your terminal to create a new account:

docker run --rm -ti --network clowder_clowder -e \
   FIRST_NAME=Admin -e LAST_NAME=User \
   -e EMAIL_ADDRESS=admin@example.com -e PASSWORD=catsarecute \
   -e ADMIN=true clowder/mongo-init

Optionally, edit these properties to your liking:

  • FIRST_NAME

  • LAST_NAME

  • EMAIL_ADDRESS

  • PASSWORD

  • ADMIN (only set this if you want the user to have superadmin rights, make sure at lease one user has this).

✅ Configuration complete! Now you can login to Clowder via localhost:9000 in your browser.

Warning

If you renamed the base clowder folder to something else, like kitten, then the --netowrk parameter must be changed to --network kitten_clowder.

Skip to using default extractors and developer resources 👇

(Optional) User creation method 2: mock SMTP server

Enable local email verification

For local instances of Clowder, the email verification step will have to be done manually, via a mock SMTP email server.

Add the following lines to the bottom of application.conf:

# application.conf

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Local email verification -- see Intellij's run console to complete registration
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
smtp.mock=true

All accounts must also be activated by an administrator. To activate your account by default, edit application.conf:

# application.conf
# Search for this line, and EDIT it (do not add a new line)
# Set to false

# Whether emails for new users registrations go through admins first
registerThroughAdmins=false

Now, create a local Clowder account via the web interface

Start Clowder:

  1. Start required services (via docker-compose up -d from the root Clowder directory).

    1. You can check if your services are already running using docker ps and check that 3 containers are active (MongoDB, ElasticSearch, and RabbitMQ) by looking at Server Containser: 3. Or check via the Docker Desktop GUI.

  2. Ensure your local clowder instance is running (on localhost:9000)

Finally, attempt to signup for an account via the Clowder GUI on localhost:9000

  • Click the Sign Up button in the top right.

Upon clicking Signup, the IntelliJ console will show the text of the user signup verification emails, where you can click the confirmation link.

Look for this in Intellij’s run output terminal, and click the link to complete registration:

<p>Please follow this
    <a href="http://localhost:9000/signup/baf28c54-80fe-480c-b1e4-9200668cb92e">link</a> to complete your registration
    at <a href="http://localhost:9000/">Clowder</a>.
</p>
  • Don’t see it? Make sure you enabled smtp.mock=true above.

Now fill in your account details, and you should be good to go using Clowder!

(Optional) Edit user properties directly in MongoDB

To edit the permissions on existing accounts, edit their properties in MongoDB. You can skip this step if you haven’t created a local Clowder account yet.

  1. Download a GUI for MongoDB: MongoDB Compass or a 3rd party tool like RoboMongo.

  2. Ensure all services are running!

cd clowder # base directory

# start all required services
docker-compose up -d
  1. Connect RoboMongo to the docker instance (the defaults should be fine)

    1. Point it towards port 27017

  2. To find user properties, in the file tree on the left, navigate to clowder → Collections → social.users

    1. Then click the dropdown to expand that user

    2. Find status field, and right click to edit.

    3. If it is Inactive, change it by typing Active (capitalized).

  3. User is activated. Refresh your browser (on localhost:9000) to access Clowder.

If Clowder feels slow, add the faster JVM option

Use the default extractors

The default extractors offer simple quality of life improvements for image, video, pdf, and audio file previews while browsing Clowder. Extractors also offer powerful capabilities for manipulating your data in Clowder, see PyClowder for additional capabilities, including running machine learning training on your data stored in Clowder.

Enable the default Extractors:

# start Clowder with extractor support
docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.extractors.yml up -d

Or run NCSA GeoServer for viewing and editing geospacial data via docker-compose.geoserver.yml:

  • geoserver

  • ncsa_geo_shp

  • extractor-geotiff-preview

  • extractor-geotiff-metadata

# start Clowder with default extractors, and GeoServer extractors
docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.extractors.yml -f docker-compose.geoserver.yml up -d

Learn more about GeoServer and read the documentation.

Troubleshooting extractors

First, make sure you include the extractors when starting Docker! See docker-compose.extractors.yml in the section above.

If running the extractor results in a "Failed to establish a new connection: [Errno 111] Connection refused", this is a Docker networking issue. Containers must be able to talk to each other (Clowder talking to RabbitMQ).

To resolve, open clowder/conf/application.conf search for and set the RabbitMQ message queue URL:

clowder.rabbitmq.clowderurl="http://host.docker.internal:9000"

Simply saving the file should fix the issue. You can again “submit a file for extraction” on the file’s details page. Done!

If navigating to localhost:9000 yields nothing, try this. On Windows, I’ve had trouble getting localhost to resolve to the Docker host, so:

  • Access Clowder not via localhost, but via your local IP address. For example, 55.251.130.193:9000.

  • Find your local IP address:
    • Windows: Settings -> Network & internet -> IPv4 address.

    • Mac: System Preferences –> Netowrk–> Advanced–> TCP/IP–> IPv4 Address. (Note: don’t use the ‘Public IP’ from iStat Menus).

    • Linux $ ifconfig

That should resolve extractor issues across all major platforms, including Apple Silicon (M1).

Next Steps

Watch the Clowder Conference playlist on Youtube!

Try the default extractors for simple quality of life improvements in Clowder.

docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.extractors.yml up -d

Write your own extractors using the PyClowder Python package.

🤔❓ Please ask any questions on our Clowder Slack.

Resources for Developers

Can’t find what you need? Clowder’s legacy wiki may have additional detail.

Requirements Overview

Following is a list of requirements for the Clowder software. Besides Java, all other services/software can be installed on other machines with Clowder configured to communicate with them.

  • Java 8 - required

    • The Clowder software is written in Scala and javascript and requires Java to execute.

    • Clowder has been tested with the OpenJDK.

    • Versions beyond 8 have not been tested.

  • MongoDB v3.4 - required

    • By default Clowder uses MongoDB to store most of the information within the system.

    • Versions above 3.4 have not been tested.

  • RabbitMQ (latest version) - optional

    • RabbitMQ is used to communicate between Clowder and the extractors. When deploying extractors it is required to deploy RabbitMQ as well.

  • ElasticSearch 2.x - optional

    • ElasticSearch is used for text based search by Clowder.

    • Versions above 2.x have not been tested.

    • This dependency (specifically v2) is not compatible with Apple Silicon M1.