Common Issues

​

Overview

​

Installation Issues

​

License Key Invalid

• License validation failed
• Invalid license key
• Services fail to start

echo $LICENSE_KEY | wc -c

curl -H "Authorization: Bearer ${LICENSE_KEY}" https://console-api.smallest.ai/validate

​

Image Pull Failed

• ImagePullBackOff
• unauthorized: authentication required
• manifest unknown

docker login quay.io
Username: your-username
Password: your-password

docker pull quay.io/smallestinc/lightning-asr:latest

kubectl get secrets -n smallest | grep registry

kubectl create secret docker-registry registry-secret \
  --docker-server=quay.io \
  --docker-username=your-username \
  --docker-password=your-password \
  --docker-email=your-email \
  -n smallest

​

Model Download Failed

• Lightning ASR stuck at startup
• Failed to download model
• Connection timeout

1. Verify URL:
   Copy

   curl -I $MODEL_URL
   
2. Check disk space:
   Copy

   df -h
   
   Need at least 30 GB free
3. Test network:
   Copy

   wget --spider $MODEL_URL
   
4. Increase timeout (Kubernetes):
   Copy

   lightningAsr:
     env:
       - name: DOWNLOAD_TIMEOUT
         value: "3600"
   

​

Runtime Issues

​

High Latency

• Requests taking >1 second
• Slow transcription
• Timeouts

nvidia-smi
kubectl exec -it <lightning-asr-pod> -- nvidia-smi

kubectl describe pod <pod-name> | grep -A5 "Limits"

lightningAsr:
  resources:
    limits:
      memory: 16Gi
      cpu: 8

sudo nvidia-smi -pm 1

gpu-operator:
  driver:
    env:
      - name: NVIDIA_DRIVER_CAPABILITIES
        value: "compute,utility"

​

Out of Memory

• Pod killed (exit code 137)
• OOMKilled status
• Memory errors in logs

1. Increase memory limit:
   Copy

   lightningAsr:
     resources:
       limits:
         memory: 20Gi
       requests:
         memory: 16Gi
   
2. Check memory leaks:
   Copy

   kubectl top pod <pod-name>
   
3. Restart pod:
   Copy

   kubectl delete pod <pod-name>
   

​

Connection Refused

• Cannot connect to API
• Connection refused
• Service unavailable

kubectl get pods -n smallest
docker compose ps

kubectl get endpoints -n smallest
curl http://localhost:7100/health

sudo iptables -L
netstat -tuln | grep 7100

​

Performance Issues

​

Slow Autoscaling

• HPA not scaling fast enough
• Pods stuck in Pending
• Cluster Autoscaler delayed

1. Reduce HPA stabilization:
   Copy

   scaling:
     auto:
       lightningAsr:
         hpa:
           scaleUpStabilizationWindowSeconds: 0
   
2. Check metrics available:
   Copy

   kubectl get hpa
   kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
   
3. Verify node capacity:
   Copy

   kubectl describe nodes | grep -A5 "Allocated resources"
   

​

Request Queue Building Up

• Increasing active requests
• Users experiencing delays
• HPA shows high metrics

1. Manual scale up:
   Copy

   kubectl scale deployment lightning-asr --replicas=10
   
2. Check autoscaling limits:
   Copy

   scaling:
     auto:
       lightningAsr:
         hpa:
           maxReplicas: 20
   
3. Add cluster capacity:
   Copy

   eksctl scale nodegroup --cluster=smallest-cluster --name=gpu-nodes --nodes=5
   

​

Data Issues

​

Transcription Quality Poor

• Low confidence scores
• Incorrect transcriptions
• Missing words

1. Check audio quality:
   • Sample rate: 16 kHz minimum (44.1 kHz recommended)
   • Format: WAV or FLAC preferred
   • Channels: Mono for best results
2. Enable punctuation:
   Copy

   {
     "url": "...",
     "punctuate": true,
     "language": "en"
   }
   
3. Verify correct language:
   Copy

   {
     "url": "...",
     "language": "es"
   }
   

​

Missing Timestamps

• No word-level timing data
• Unable to sync with video

{
  "url": "...",
  "timestamps": true
}

{
  "words": [
    {"word": "Hello", "start": 0.0, "end": 0.5}
  ]
}

​

Network Issues

​

Cannot Reach License Server

• Grace period activated
• Connection to license server failed
• Services still working but warnings

1. Test connectivity:
   Copy

   curl -v https://console-api.smallest.ai
   
2. Check firewall rules:
   • Allow outbound HTTPS (port 443)
   • Whitelist console-api.smallest.ai
3. Review network policies (Kubernetes):
   Copy

   kubectl get networkpolicy -n smallest
   
4. Monitor grace period:
   Copy

   kubectl logs -l app=license-proxy | grep -i "grace"
   

​

Slow Downloads

• Model download taking >30 minutes
• Audio file upload slow

1. Use faster network:
   • AWS S3 in same region
   • CloudFront CDN
2. Enable parallel downloads:
   Copy

   lightningAsr:
     env:
       - name: DOWNLOAD_WORKERS
         value: "4"
   
3. Use shared storage (Kubernetes):
   Copy

   models:
     volumes:
       aws:
         efs:
           enabled: true
   

​

Quick Diagnostics

​

One-Command Health Check

curl http://localhost:7100/health && \
  kubectl get pods -n smallest && \
  kubectl top nodes && \
  kubectl top pods -n smallest

​

Collect All Logs

kubectl logs -l app=lightning-asr -n smallest --tail=100 > asr-logs.txt
kubectl logs -l app=api-server -n smallest --tail=100 > api-logs.txt
kubectl logs -l app=license-proxy -n smallest --tail=100 > license-logs.txt

​

Test Transcription

curl -X POST http://localhost:7100/v1/listen \
  -H "Authorization: Token ${LICENSE_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www2.cs.uic.edu/~i101/SoundFiles/StarWars60.wav"}'

​

Getting Help