Analyze Images and PDFs with Google Gemini's Multimodal API in Python

What you'll build

A Python script that sends a local JPEG and a PDF to Gemini 1.5 Flash and parses a structured JSON object from each response, using Google's official google-generativeai SDK.

Prerequisites

• Python 3.9 or newer (python --version to check)
• A Google AI Studio API key — free at aistudio.google.com/app/apikey
• pip 23+
• A sample JPEG and a sample PDF on disk

OS note: Commands below use export (macOS/Linux). On Windows PowerShell use $env:GEMINI_API_KEY = "your-key".

Step 1 — Store your API key safely

Never hard-code credentials. Export the key as an environment variable in your terminal session:

export GEMINI_API_KEY="your-key-here"

Step 2 — Install dependencies

pip install "google-generativeai>=0.7.0" Pillow

Pillow loads local images; the Google SDK handles everything else, including PDF uploads via the File API.

Step 3 — Write the script

Create gemini_multimodal.py:

Advertisement

Shadow GPS — know where it is, always Real-time GPS tracking for vehicles, gear and loved ones. No monthly contracts.

import json
import os
import time

import PIL.Image
import google.generativeai as genai

# Configure the SDK (reads key from the environment — never commit secrets)
genai.configure(api_key=os.environ["GEMINI_API_KEY"])

# Setting response_mime_type tells Gemini to emit valid JSON every time
model = genai.GenerativeModel(
    model_name="gemini-1.5-flash",
    generation_config={"response_mime_type": "application/json"},
)


def analyze_image(path: str, prompt: str) -> dict:
    """Open a local image (JPEG/PNG/WebP) and return parsed JSON."""
    image = PIL.Image.open(path)
    response = model.generate_content([image, prompt])
    return json.loads(response.text)


def analyze_pdf(path: str, prompt: str) -> dict:
    """Upload a PDF via the File API, query it, then delete the upload."""
    uploaded = genai.upload_file(path)

    # Wait for Google's servers to finish processing (usually instant for small files)
    while uploaded.state.name == "PROCESSING":
        time.sleep(2)
        uploaded = genai.get_file(uploaded.name)

    response = model.generate_content([uploaded, prompt])
    genai.delete_file(uploaded.name)   # optional: remove from Google servers immediately
    return json.loads(response.text)


if __name__ == "__main__":
    # --- Image ---
    img_result = analyze_image(
        "sample.jpg",
        "Return JSON with keys: description, dominant_colors (list), has_people (bool).",
    )
    print("IMAGE:", json.dumps(img_result, indent=2))

    # --- PDF ---
    pdf_result = analyze_pdf(
        "document.pdf",
        "Return JSON with keys: title, summary (one sentence), page_count_estimate (int).",
    )
    print("PDF:", json.dumps(pdf_result, indent=2))

Why response_mime_type? Without it, the model sometimes wraps JSON in markdown fences (```json … ```), breaking json.loads. This config key forces clean JSON output at the model level.

Why upload_file for PDFs? PDFs can't be opened with Pillow. The File API accepts application/pdf up to 2 GB and stores the file for up to 48 hours.

Step 4 — Run it

Place sample.jpg and document.pdf in the same directory, then:

python gemini_multimodal.py

Verify it works

Expected output shape (values vary by file):

IMAGE: {
  "description": "A golden retriever sitting in a sunny park.",
  "dominant_colors": ["yellow", "green", "blue"],
  "has_people": false
}
PDF: {
  "title": "Q3 Financial Report",
  "summary": "Revenue grew 12 % year-over-year driven by cloud services.",
  "page_count_estimate": 8
}

Both blocks must parse without error via json.loads. If the script exits without exceptions, you're done.

Troubleshooting

Next steps

• Typed schemas — Pass a TypedDict or Pydantic model as response_schema inside GenerationConfig (SDK ≥ 0.8) to validate field types automatically.
• Multi-image comparison — Pass a list: [img1, img2, "Compare these two images"].
• Vertex AI — Replace google-generativeai with the vertexai SDK for IAM auth, VPC controls, and enterprise quotas.
• Official vision docs: ai.google.dev/gemini-api/docs/vision