• Deploy autogpt on Jina AI Cloud with one command

  lc-serve deploy autogpt
  

  Show command output

  ╭──────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ App ID       │                                           autogpt-6cbd489454                                           │
  ├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Phase        │                                                Serving                                                 │
  ├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Endpoint     │                                 wss://autogpt-6cbd489454.wolf.jina.ai                                  │
  ├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ App logs     │                                        dashboards.wolf.jina.ai                                         │
  ├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Swagger UI   │                              https://autogpt-6cbd489454.wolf.jina.ai/docs                              │
  ├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ OpenAPI JSON │                          https://autogpt-6cbd489454.wolf.jina.ai/openapi.json                          │
  ╰──────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────╯
  

• Integrate autogpt with external services using APIs. Get a flavor of the integration on your CLI with

  lc-serve playground autogpt
  

  Show playground

  

• Deploy babyagi on Jina AI Cloud with one command

  lc-serve deploy babyagi
  

• Integrate babyagi with external services using our Websocket API. Get a flavor of the integration on your CLI with

  lc-serve playground babyagi
  

  Show playground

  

• Deploy pandas-ai on Jina AI Cloud

  lc-serve deploy pandas-ai
  

  Show command output

  ╭──────────────┬─────────────────────────────────────────────────────────────────────────────────╮
  │ App ID       │                               pandasai-06879349ca                               │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ Phase        │                                     Serving                                     │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ Endpoint     │                     wss://pandasai-06879349ca.wolf.jina.ai                      │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ App logs     │                             dashboards.wolf.jina.ai                             │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ Swagger UI   │                  https://pandasai-06879349ca.wolf.jina.ai/docs                  │
  ├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ OpenAPI JSON │              https://pandasai-06879349ca.wolf.jina.ai/openapi.json              │
  ╰──────────────┴─────────────────────────────────────────────────────────────────────────────────╯
  

• Upload your DataFrame to Jina AI Cloud (Optional - you can also use a publicly available CSV)

  • Define your DataFrame in a Python file

    # dataframe.py
    import pandas as pd
    df = pd.DataFrame(some_data)
    

  • Upload your DataFrame to Jina AI Cloud using <module>:<variable> syntax

    lc-serve util upload-df dataframe:df
    

• Conversationalize your DataFrame using pandas-ai APIs. Get a flavor of the integration with a local playground on your CLI with

  lc-serve playground pandas-ai <host>
  

  Show playground

  

• Deploy pdf_qna on Jina AI Cloud with one command

  lc-serve deploy pdf-qna
  

• Get a flavor of the integration with Streamlit playground on your CLI with

  lc-serve playground pdf-qna
  

  Show playground

  

• Expand the Q&A bot to multiple languages, different document types & integrate with external services using simple REST APIs.

  https://github.com/jina-ai/langchain-serve/blob/8f7a9272e99490a5357655becfc5da3569655f38/lcserve/apps/pdf_qna/app.py#L8-L12

from lcserve import serving

def authorizer(token: str) -> Any:
    if not token == 'mysecrettoken':            # Change this to add your own authorization logic
        raise Exception('Unauthorized')         # Raise an exception if the request is not authorized

    return 'userid'                             # Return any user id or object

@serving(auth=authorizer)
def ask(question: str, **kwargs) -> str:
    auth_response = kwargs['auth_response']     # This will be 'userid'
    return ...

@serving(websocket=True, auth=authorizer)
async def talk(question: str, **kwargs) -> str:
    auth_response = kwargs['auth_response']     # This will be 'userid'
    return ...

🤔 Gotchas about the auth function

• Should accept only one argument token.
• Should raise an Exception if the request is not authorized.
• Can return any object, which will be passed to the auth_response object under kwargs to the functions.
• Expects Bearer token in the Authorization header of the request.
• Sample HTTP request with curl:

  curl -X 'POST' 'http://localhost:8080/ask' -H 'Authorization: Bearer mysecrettoken' -d '{ "question": "...", "envs": {} }'
  

• Sample WebSocket request with wscat:

  wscat -H "Authorization: Bearer mysecrettoken" -c ws://localhost:8080/talk
  

from lcserve import serving

@serving
def store(text: str, **kwargs):
    workspace: str = kwargs.get('workspace')
    path = f'{workspace}/store.txt'
    print(f'Writing to {path}')
    with open(path, 'a') as f:
        f.writelines(text + '\n')
    return 'OK'


@serving(websocket=True)
async def stream(**kwargs):
    workspace: str = kwargs.get('workspace')
    websocket: WebSocket = kwargs.get('websocket')
    path = f'{workspace}/store.txt'
    print(f'Streaming {path}')
    async with aiofiles.open(path, 'r') as f:
        async for line in f:
            await websocket.send_text(line)
    return 'OK'

Here, we are using the workspace to store the incoming text in a file via the REST endpoint and streaming the contents of the file via the WebSocket endpoint.

Let's take an example of a simple FastAPI app with directory structure

.
└── endpoints.py

# endpoints.py
from typing import Union

from fastapi import FastAPI

app = FastAPI()


@app.get("/status")
def read_root():
    return {"Hello": "World"}


@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}

lc-serve deploy jcloud --app endpoints:app

Let's take an example of a simple app that uses OPENAI_API_KEY stored as secrets.

This app directory contains the following files:

.
├── main.py             # The app
├── jcloud.yml          # JCloud deployment config file
├── README.md           # This README file
├── requirements.txt    # The requirements file for the app
└── secrets.env         # The secrets file containing the redis credentials

Note secret.env in this directory is a dummy file. You should replace it with your own secrets after creating a Redis instance. (For example with Upstash), such as:

OPENAI_API_KEY=sk-xxx

main.py will look like:

# main.py
from lcserve import serving
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from langchain.chat_models import ChatOpenAI

prompt = PromptTemplate(
    input_variables=["subject"],
    template="Write me a short poem about {subject}?",
)


@serving(openai_tracing=True)
def poem(subject: str, **kwargs):
    tracing_handler = kwargs.get("tracing_handler")

    chat = ChatOpenAI(temperature=0.5, callbacks=[tracing_handler])
    chain = LLMChain(llm=chat, prompt=prompt, callbacks=[tracing_handler])

    return chain.run(subject)

In the above example, the app will use OPENAI_API_KEY provided by the secrets to interact with OpenAI.

Then you can deploy using the following command and interact with the deployed endpoint.

lc-serve deploy jcloud main --secrets secrets.env

Consider an HTTP application that has served requests for 10 minutes in the last hour and uses a custom config:

instance: C4
autoscale_min: 0
disk_size: 2G

Total credits per hour charged would be 3.33. The calculation is as follows:

C4 instance has an hourly credit rate of 20.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 0 + 2 * 0.104 = 0.208 (since `autoscale_min` is 0)
Serving credits = 20 * 10/60 = 3.33
Total credits per hour = 0.208 + 3.33 = 3.538

Consider a WebSocket application that had active connections for 20 minutes in the last hour and uses the default configuration.

instance: C3
autoscale_min: 1
disk_size: 1G

Total credits per hour charged would be 13.33. The calculation is as follows:

C3 instance has an hourly credit rate of 10.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 10 + 1 * 0.104 = 10.104 (since `autoscale_min` is 1)
Serving credits = 10 * 20/60 = 3.33
Total credits per hour = 10.104 + 3.33 = 13.434

lc-serve command is registered during langchain-serve installation. If you get command not found: lc-serve error, please replace lc-serve command with python -m lcserve & retry.

If you make long HTTP/ WebSocket requests, the default timeout value (2 minutes) might not be suitable for your use case. You can provide a custom timeout value during JCloud deployment by using the --timeout argument.

Additionally, for HTTP, you may also experience timeouts due to limitations in the OSS we used in langchain-serve. While we are working to permanently address this issue, we recommend using HTTP/1.1 in your client as a temporary workaround.

For WebSocket, please note that the connection will be closed if idle for more than 5 minutes.

We provide 2 options to pass environment variables:

1. Use --env during app deployment to load env variables from a .env file. For example, lc-serve deploy jcloud app --env some.env will load all env variables from some.env file and pass them to the app. These env variables will be available in the app as os.environ['ENV_VAR_NAME'].

2. You can also pass env variables while sending requests to the app both in HTTP and WebSocket. envs field in the request body is used to pass env variables. For example

   {
       "question": "What is the meaning of life?",
       "envs": {
           "ENV_VAR_NAME": "ENV_VAR_VALUE"
       }
   }
   

Please use --verbose and retry to get more information. If you are operating on computer with arm64 arch, please retry with --platform linux/amd64 so the image can be built correctly.

```bash
textual console -x EVENT -x SYSTEM -x DEBUG
```

2. Start the playground with --verbose flag. Start interacting and see the logs in the console.

```bash
lc-serve playground babyagi --verbose
```