Uni-Code: Leveraging Monorepo for Enhanced Collaboration and Efficiency

Introduction

In the software development process, the way codes are organized directly impacts team collaboration efficiency. There are two mainstream code management strategies: Multi-repo and Monorepo. In this article, we will discuss how we use a Monorepo to replace the original multi-repository approach for Tencent Cloud Operating System.

What is Monorepo?

A Monorepo is a project management strategy that allows multiple projects or modules to be stored within a single shared code repository, whereas Multi-repo means each project or module resides in its own separate repository.

Why do we need to use Monorepo in our business?

Before discussing Monorepo, it's important to understand some Multi-repo's shortcomings that Monorepo aims to address.

Shortcomings of Multi-repo:

1. Code Reuse Sometimes Causes Trouble: In our case, we had similar apps deployed on different platforms, with a lot of repetitive codes.

   

   An enum definition can repeat 10 times, including many outdated definitions.

   When a common module changed, updating all modules that depend on it became a cumbersome task.

   Additionally, because code was dispersed across various locations, searching through repositories often added considerable pain to the team.

2. Complex Publish Management:

   In a multi-repo environment, changes to each project or module are recorded independently, making it exceptionally difficult to understand the entire system's change history.

   Automation tools struggle to link changes across repositories, thereby affecting the efficiency of version management and release processes.

Advantages of Using Monorepo:

Now, Let's Establishing an Efficient Monorepo for Business

Challenges

While Monorepo offers numerous advantages, fully unleashing its potential requires careful implementation. Here were some primary challenges our business faced:

Fundamental Challenges:

1. Dependency Tool Decision: Our previous shared package was using yarn + lerna. Should we stick with it, or shift to pnpm?

2. Cross-Project Modules Management: When modules are used across front-end and back-end projects, it's vital to design a robust package structure. Creating a shared package that works seamlessly across various environments (e.g., Web, mini-apps, Node.js, Vite, Webpack) and addresses compatibility issues.

Advanced Challenges:

3. Changelog Management: If each project has it's own changelog and independent pipelines, there will be huge management and maintenance costs. Therefore, standardized Changelog and standard pipeline are needed .

4. Server-Side Rendering (SSR) Support

5. Adopting Mixed-Language Development (TypeScript/Rust): Using multiple programming languages ​​(such as TypeScript and Rust) is an effective way to improve performance. Can we find a way to organize and manage code in different languages ​​in Monorepo?

Let’s talk about the solutions to each problem one by one:

I. Dependency Tool Selection:

Conclusion: Choose pnpm

Considering the significant advantages of pnpm in terms of dependency management efficiency, security, and natural support for Monorepo, we have concluded that choosing pnpm as our dependency management tool is the superior decision.

II. Cross-Project Modules Management:

2.1 Module System Selection

The evolution of JavaScript module systems can be summarized as follows: CommonJS (CJS) → Asynchronous Module Definition (AMD) → Common Module Definition (CMD) → ECMAScript Modules (ESM).

Currently, AMD and CMD are relatively complex and lack sufficient support from build tools, making them less popular module systems.

When designing a monorepo, it is essential to choose the appropriate module system to ensure seamless code integration across applications. Here, we will compare various module bundling methods to determine the best fit for our monorepo structure.

After understanding these module types, the best practices for choosing module specifications are as follows:

• Shared Configuration Modules (Cross Frontend and Backend): Publish packages in both ESM and CJS formats. While ESM is the modern choice, CJS remains a fallback option due to the ongoing improvements in Node.js support for ESM.

• UI Component Libraries: Publish packages in both ESM and CJS formats. This ensures that the component libraries can be used in various development environments, whether in frontend frameworks like React or Vue or with bundling tools like Webpack or Rollup.

• For Node.js Projects: Currently, only publish packages in CJS format. CJS is the default module system in Node.js, offering stable and extensive support, making it suitable for most backend project needs.

• Compatibility: Ensure that the modules run seamlessly in the target runtime environment. Projects that need to support older browsers or Node.js versions may require additional transpilation steps (such as using Babel).

2.2 Package Structure Design

We need to design a monorepo that supports backend projects, frontend component libraries, frontend projects, and cross-frontend and backend libraries. Therefore, we have designed the directory structure as follows:

/monorepo
│
├── /apps/                   # Frontend and backend applications
│   ├── /client/             # Frontend applications
│   │   ├── next-app         # Next.js project(a React framework with server-side rendering capabilities)
│   │   ├── react-app        # React project
│   │   ├── uni-app          # uni-app (a vue-based framework)
│   │   ├── vite-app         # Vite project
│   │   ├── vite-react-app   # Vite and React project
│   │
│   ├── /service/            # Backend services
│       ├── deno-app         # Backend project using Deno (a modern JavaScript/TypeScript runtime)
│       ├── node-app         # Backend project using Node.js
│
├── /packages/               # Shared libraries
│   ├── /config/             # Shared configuration modules
│   ├── /utils/              # Shared utility modules
│   │   ├── /native/         # Native modules (Rust/C++)
│   │   ├── /js/             # js utils shared across front-end and back-end projects
│   │   ├── /tests/          # Test code
│   │   ├── package.json     # Project configuration file
│   │
│   ├── /ui/                 # Component libraries
│   │   ├── /react/          # React components
│   │   ├── /vue/            # Vue components
│
├── /scripts/                # Scripts
│   ├── build.js             # Build script
│   ├── deploy.js            # Deployment script
│   └── ...
│
├── package.json             # Root-level project configuration file
├── pnpm-workspace.yaml      # PNPM workspace configuration file

We have also established a CLI tool: a command-line interface based on Commander + Inquirer + Chalk. The related commands are already written into the custom command set of the monorepo. Running pnpm create initializes the setup, and there are corresponding templates for common library projects, UI library projects, and frontend/backend seed repositories.

Inquirer.js is a Node.js library that makes it easy to create interactive command-line prompts.

Chalk is a Node.js library that allows you to style text in the command-line.

2.3 Designing a Shared Package

Now, we will cover how to configure and package shared modules or application projects in a Monorepo. We'll start with shared modules and then move to an example application, such as a React project.

2.3.1. Shared Modules

When packaging cross-platform utilities, we should support both CommonJS (CJS) and ES Modules (ESM), as concluded previously. In our project, we use tsup for its simplicity and speed.

Tsup Configuration Example:

// packages/utils/tsup.config.ts
import { defineConfig } from 'tsup';

export default defineConfig({
  entry: ['utils/index.ts', 'types/index.ts'],
  clean: true,
  dts: true,
  outDir: 'dist',
  format: ['cjs', 'esm']
});

package.json Configuration for utilities:

{
  "name": "@infras/shared",
  "version": "0.0.1",
  "description": "Shared utilities for all projects and apps.",
  "exports": {
    "./utils": {
      "import": "./dist/utils/index.mjs",
      "require": "./dist/utils/index.js"
    },
    "./types": {
      "import": "./dist/types/index.mjs",
      "require": "./dist/types/index.js"
    }
  },
  "main": "dist/index.js",
  "module": "dist/index.mjs",
  "typings": "dist/index.d.ts",
  "sideEffects": false,
  "scripts": {
    "prepare": "npm run build",
    "dev": "tsup --watch",
    "build": "tsup"
  },
  "devDependencies": {
    "tsup": "^5.10.3"
  }
}

2.3.2. Packaging UI Component Library

For a UI component library, the packaging process is similar to utilities. We need to support both CJS and ESM formats and ensure TypeScript definitions are included.

Tsup Configuration for UI Components:

// packages/ui/tsup.config.ts
import { defineConfig } from 'tsup';

export default defineConfig({
  entry: ['components/index.ts'],
  clean: true,
  dts: true,
  outDir: 'dist',
  format: ['cjs', 'esm']
});

package.json Configuration for UI Components:

{
  "name": "@infras/ui",
  "version": "0.0.1",
  "description": "UI component library for all projects and apps.",
  "exports": {
    "./components": {
      "import": "./dist/components/index.mjs",
      "require": "./dist/components/index.js"
    }
  },
  "main": "dist/index.js",
  "module": "dist/index.mjs",
  "typings": "dist/index.d.ts",
  "sideEffects": false,
  "scripts": {
    "prepare": "npm run build",
    "dev": "tsup --watch",
    "build": "tsup"
  },
  "devDependencies": {
    "tsup": "^5.10.3"
  }
}

2.4 Application Example: React Project

Packaging and Configuring package.json

For the application, such as a React project, we'll configure the package.json to include the necessary dependencies and scripts.

{
  "name": "react-app",
  "version": "1.0.0",
  "private": true,
  "dependencies": {
    "react": "^17.0.0",
    "react-dom": "^17.0.0",
    "@infras/shared": "workspace:*",
    "@infras/ui": "workspace:*"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "devDependencies": {
    "react-scripts": "^4.0.3"
  }
}

2.4.2. Importing Shared Packages in React Project

In the React application, you can import the shared utilities and UI components as follows:

// src/index.tsx
import React from 'react';
import ReactDOM from 'react-dom';
import { AppType } from '@infras/shared/types';
import { sum } from '@infras/shared/utils';
import { Button } from '@infras/ui/components';

const App = () => (
  <div>
    <h1>App Type: {AppType.Web}</h1>
    <p>Sum: {sum(1, 2)}</p>
    <Button>Click Me</Button>
  </div>
);

ReactDOM.render(<App />, document.getElementById('root'));

2.5 Additional Considerations 👀

Workspace Configuration:

We use PNPM, which natively supports workspaces. A workspace is a set of related projects managed within one repository and benefits seamless integration between them. PNPM workspace allows easy linking of workspace projects, directly depending on the code level, rather than having to use package management / cross-repository dependencies. No need for version management. Example pnpm-workspace.yaml:

# This YAML file configures the workspace for PNPM.
# It specifies the locations of the packages within the monorepo.
packages:
  - 'apps/*'
  - 'packages/*'

Shared Dependencies:

Define shared dependencies at the root level.

Root package.json:

{
  "name": "monorepo",
  "private": true,
  "workspaces": [
    "apps/*",
    "packages/*"
  ],
  "dependencies": {
    "react": "^17.0.0",
    "react-dom": "^17.0.0"
  },
  "devDependencies": {
    "eslint": "^8.5.0",
    "prettier": "^2.3.2"
  }
}

For open collaboration on shared packages: it is more convenient to actively submit code for projects of other groups, such as fixing defects or implementing new features we need. However, to avoid issues brought by open collaboration,

Code modifications should consider backward compatibility as much as possible After the modification is completed, automatically run unit tests of the relevant caller When merging requests, identify modifications to the relevant public library and automatically set the corresponding person in charge as a necessary reviewer Notify the testing team to conduct relevant system tests and core regression tests

2.6 How to Run:

• React project: pnpm start --filter "react-app"

• Node.js project: pnpm start --filter "node-app"

• Vite project: pnpm start --filter "vite-app"

III. Changelog Management:

Managing changelogs in a monorepo can be challenging due to the complexity of handling multiple projects with independent update cycles.

We chose changesets to tackle this challenge. changesets is recommended by pnpm, and after our investigation, we discovered that the TikTok FE team also relies on it.

Here's how changesets works:

Each change creates a new Changeset to describe the modifications made to a particular package, thus managing version numbers.

When a package fixes a bug, changesets records the bug fix and updates the package's version to a new release. If other packages depend on this package, changesets records this version change in the Changeset of these dependent packages and updates their version numbers accordingly.

If multiple Changesets simultaneously modify the dependencies of the same package, changesets automatically merges these changes and ensures that the final dependency relationship remains valid.

Below is a brief introduction to how we use Changesets with PNPM.

Step 1: Install Changesets

Install Changesets as a dev dependency in your monorepo

# in the root directory
pnpm add @changesets/cli -D

Step 2: Initialize Changesets

Initialize Changesets in your repository,create a .changeset folder in the root of your repository with some configuration files.

pnpm changeset init

Step 3: Configure Changesets

Edit the .changeset/config.json file

{
  "changelog": ["@changesets/changelog-github", { "repo": "your-org/your-repo" }],
  "commit": false, // Specifies whether to automatically commit changesets.
  "fixed": [], // Specifies any fixed changeset versions.
  "linked": [], // Specifies linked changeset versions.
  "access": "public", // The access level for published packages.
  "baseBranch": "main", // The base branch for version bumps.
  "updateInternalDependencies": "patch", // Specifies how to update internal dependencies.
  "ignore": [] // Specifies any files to ignore when creating changesets.
}

Step 4: Versioning and Publishing

Now you can develop your code normally and commit your changes. When you are ready to release a new version, run the following command to bump versions and update the changelog:

pnpm changeset version

When the pnpm changeset publish command is run, the changeset tool will compare the current state of the my-package package with the previous version stored in the version control system. If any changes are detected, the changeset tool will increment the version of the package based on the type of changes made

the version of a package is updated based on the type of changes made to the package. For example, if a change is considered a "breaking" change, the version of the package would be updated to a new major version.

This command will:

1. Update the versions of the changed packages.
2. Generate a new changelog entry.
3. Commit the changes.

After versioning, you can publish the packages to the registry:

pnpm publish -r

Step 6: Automating with CI/CD

You can automate the versioning and publishing process using CI/CD tools ，Here’s an example workflow file (.github/workflows/release.yml) for GitHub Actions:

name: Release

on:
  push:
    branches:
      - main

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '14'
          cache: 'pnpm'

      - name: Install dependencies
        run: pnpm install

      - name: Create and publish changeset
        run: |
          pnpm changeset version
          git config --global user.email "you@example.com"
          git config --global user.name "Your Name"
          git add .
          git commit -m "chore(release): version packages"
          git push
          pnpm publish -r
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Summary This is the process of using changesets. This method allows us to better manage the changelog in a Monorepo, ensuring version consistency and traceability.

IIII. Supporting Server-Side Rendering (SSR) in the Component Library:

Component Library Packaging

When designing and packaging a component library, we aim for it to be compatible with multiple environments, including browsers and Node.js for server-side rendering (SSR). Using @infras/ui as an example, let's first introduce the packaging and usage of the component library.

Packaging Configuration Example

1. Packaging Process:
   • Use tsup for packaging, specifying entry files and output formats (ESM and CJS).
   • The generated files are stored in the dist directory, and type definition files are generated for use in TypeScript projects.

First, we use tsup to package the library, generating both ESM and CJS formats, along with type definition files:

// packages/ui/tsup.config.ts
import { defineConfig } from 'tsup';

export default defineConfig({
  entry: ['react/index.ts', 'vue/index.ts'],
  clean: true,
  dts: true,
  outDir: 'dist',
  format: ['cjs', 'esm'],
  external: ['react', 'vue']
});

After executing tsup, the dist directory structure will be as follows:

# packages/ui
- dist
  - react
    - index.js (CJS)
    - index.mjs (ESM)
    - index.d.ts (TypeScript)
  - vue
    - index.js (CJS)
    - index.mjs (ESM)
    - index.d.ts (TypeScript)
- react
  - index.ts
- vue
  - index.ts
- package.json

package.json Configuration

The configuration for the component library's package.json is as follows:

// packages/ui/package.json
{
  "name": "@infras/ui",
  "version": "0.0.1",
  "description": "A UI component library for React and Vue.",
  "main": "dist/react/index.js",
  "module": "dist/react/index.mjs",
  "exports": {
    "./react": {
      "import": "./dist/react/index.mjs",
      "require": "./dist/react/index.js"
    },
    "./vue": {
      "import": "./dist/vue/index.mjs",
      "require": "./dist/vue/index.js"
    }
  },
  "typings": "dist/index.d.ts",
  "sideEffects": false,
  "scripts": {
    "build": "tsup"
  },
  "devDependencies": {
    "tsup": "^5.10.3"
  },
  "peerDependencies": {
    "react": "^17.0.0 || ^18.0.0",
    "vue": "^3.0.0"
  }
}

Using the Component Library in Front-End Applications

2. Usage Process:
   • In the front-end application, include the component library by configuring dependencies and dependenciesMeta.
   • Import the component library in both React and Vue applications and use the components as needed.

Importing the Component Library in React and Vue Applications

• React Application:

  // apps/react-app/index.tsx
  import { Component } from '@infras/ui/react';
  
  const App = () => <Component />;
  export default App;

• Vue Application:

  <!-- apps/vue-app/index.vue -->
  <script setup lang="ts">
  import { Component } from '@infras/ui/vue';
  </script>
  
  <template>
    <Component />
  </template>

Component Library Server-Side Rendering (SSR)

3. Server-Side Rendering:
   • Ensure the component library can run in a Node.js environment for server-side rendering.
   • For React, use react-dom/server to render components to strings and output them on the server side.
   • For Vue, use vue-server-renderer to render components to HTML strings on the server side.

• React SSR Principle:

  • Use the ReactDOMServer.renderToString method to render React components as HTML strings, then return the string on the server side.
  • This approach improves initial rendering speed and benefits SEO.

• Vue SSR Principle:

  • Vue SSR uses methods provided by vue-server-renderer to render Vue components as HTML strings.
  • Similar to React's renderToString, Vue's renderToString method renders components as HTML strings and returns them on the server side.

To support server-side rendering (SSR) in a Node.js environment, the component library must be able to run on the server and correctly render components.

• React SSR:

  // apps/node-app/index.js
  const { Component } = require('@infras/ui/react');
  const React = require('react');
  const ReactDOMServer = require('react-dom/server');
  
  console.log('SSR: ', ReactDOMServer.renderToString(React.createElement(Component)));

Building and Running the Application

• Start the React application:

  pnpm start --filter "react-app"

• Start the Vue application:

  pnpm start --filter "vite-app"

Through these steps, you can effectively utilize Rust's high-performance computing capabilities in a Monorepo project to handle complex and performance-intensive tasks.

If you have more specific needs or issues, you can further extend and customize these configurations.

V. Adopting Mixed-Language Development (TypeScript/Rust)

In a Monorepo, using npm modules written in Rust or Golang can help handle CPU-intensive tasks. Functions compiled in Rust perform significantly better than native Node.js functions. For example, a simple sum function's execution time improved from 8.44ms to 0.069ms.

Writing Native Language Modules with Rust

Project Directory Structure

# packages/rs
  - src
    - lib.rs
  - npm
  - index.js
  - index.d.ts
  - package.json
  - Cargo.toml

Initializing the Rust Module

Use napi-rs to initialize an npm module package built with Rust. napi-rs opts for the CJS format to ensure compatibility with ESM (related to node#40541), allowing it to be used without additional modifications.

package.json Configuration

Below is the package.json configuration initialized by napi-rs:

// packages/rs/package.json
{
  "name": "@infras/rs",
  "version": "0.0.0",
  "type": "commonjs",
  "main": "index.js",
  "types": "index.d.ts",
  "devDependencies": {
    "@napi-rs/cli": "^2.0.0"
  },
  "scripts": {
    "prepare": "npm run build",
    "artifacts": "napi artifacts",
    "build": "napi build --platform --release",
    "build:debug": "napi build --platform",
    "version": "napi version"
  }
}

Example Rust Code

Write Rust code in src/lib.rs, such as a simple sum function:

#[macro_use]
extern crate napi_derive;

#[napi]
fn sum(a: i32, b: i32) -> i32 {
    a + b
}

Using the Rust Module in a Node.js Application

Declare the native language module dependency in the Node.js application's package.json:

// apps/node-app/package.json
{
  "dependencies":{
     "@infras/rs": "workspace:*"
  }
}

Call the Rust module in the Node.js application:

• CommonJS Mode:

// apps/node-app/index.js
const { sum } = require('@infras/rs');
console.log('Rust `sum(1, 1)`:', sum(1, 1)); // 2

• ESM Mode:

// apps/node-app/index.mjs
import { sum } from '@infras/rs';
console.log('Rust `sum(1, 1)`:', sum(1, 1)); // 2

Building and Running

Use the commands provided by napi-rs to build the Rust module:

cd packages/rs
pnpm run build

After the build is complete, you can run and test it in the Node.js application:

pnpm start --filter "node-app"

By following these steps, you can effectively leverage Rust's high-performance computing capabilities within a Monorepo project to handle complex and performance-critical tasks.

For more specific needs or issues, these configurations can be further extended and customized.

Summary

After implementing our Monorepo, we successfully managed six operational systems by combining it with micro-frontend + monorepo. The centralized repository management significantly enhanced open collaboration within the team, making it more convenient and productive. and troubleshooting was simplified, leading to higher overall productivity and quality in our development lifecycle.

This integration also garnered consistent praise from our product teams for its streamlined processes. If you found this article useful, you can give it a star 🌟.

refernce article：

monorepo-vs-polyrepo publish-esm-and-cjs changesets-is-a-game-changer