AU2023200526B2 - Immersnet: a generative model for extrapolating 360° panoramas from narrow field of view image - Google Patents

Abstract

#$%^&*AU2023200526B220250814.pdf##### ABSTRACT Embodiments are disclosed for generating 360-degree panoramas from input narrow field of view images. A method of generating 360-degree panoramas may include obtaining an input image and guide, generating a panoramic projection of the input image, and generating, by a panorama generator, a 360-degree panorama based on the panoramic projection and the guide, wherein the panorama generator is a guided co-modulation generator network trained to generate a 360-degree panorama from the input image based on the guide. ABSTRACT Embodiments are disclosed for generating 360-degree panoramas from input narrow field of 2023200526 01 Feb 2023 view images. A method of generating 360-degree panoramas may include obtaining an input image and guide, generating a panoramic projection of the input image, and generating, by a panorama generator, a 360-degree panorama based on the panoramic projection and the guide, wherein the panorama generator is a guided co-modulation generator network trained to generate a 360-degree panorama from the input image based on the guide. FI G . 1 IN PU T 10 2 PA NO RA MI C PR OJ EC TI ON MA NA GE R 10 8 PA NO RA MA G EN ER AT OR 11 0 PA NO RA MA G EN ER AT IO N SY ST EM 10 0 1 4 2 3 EX TR AP OL AT ED PA NO RA MA 12 0 IN PU T IM AG E 10 4 GU ID E 10 6 5 1/14 20 23 20 05 26 0 1 Fe b 20 23 2023200526 01 Feb 2023 PANORAMA GENERATION SYSTEM 100 INPUT IMAGE 104 2 4 EXTRAPOLATED PANORAMIC PROJECTION PANORAMA GENERATOR 1 3 5 PANORAMA GUIDE 106 MANAGER 108 110 1/14 120 INPUT 102 FIG. 1

Description

2023200526 01 Feb 2023

100 SYSTEM GENERATION PANORAMA PANORAMA GENERATION SYSTEM 100 104 IMAGE INPUT 4

2 INPUT IMAGE 104 2 4 GENERATOR PANORAMA PROJECTION PANORAMIC EXTRAPOLATED

EXTRAPOLATED PANORAMIC PROJECTION PANORAMA GENERATOR PANORAMA

3

1 5

1 3 5 PANORAMA

MANAGER 108 110

GUIDE 106 MANAGER 108 110 1/14 1/14

GUIDE 106 120

120

INPUT 102 INPUT 102

FIG. 1

FIG. 1

IMMERSNET: AA GENERATIVE GENERATIVE MODEL MODELFOR FOR EXTRAPOLATING 360 PANORAMAS PANORAMAS 2023200526 01 Feb 2023

IMMERSNET: EXTRAPOLATING 360° FROMNARROW FROM NARROW FIELDOF FIELD OFVIEW VIEWIMAGE IMAGE

BACKGROUND BACKGROUND

[0001]

[0001] Photographsshow Photographs show a glimpse a glimpse of realitycaptured of reality captured when when the the shutter shutter is is pressed:they pressed: they

are are but but aa window window onona amoment moment frozen frozen in time. in time. Yet, Yet, despite despite thethe limitsofofcameras, limits cameras,oneone cancan easily easily

imagine the scene imagine the sceneininwhich whichthethe image image was was captured captured by picturing by picturing the contents the contents surrounding surrounding the the

photograph: surely photograph: surely it it was was aa large large tree tree that that was casting this was casting this shadow onthe shadow on thelawn, lawn,and andthere therewere were

undoubtedlyother undoubtedly othervehicles vehiclesandand pedestrians pedestrians passing passing by this by this busy busy street. street. In computer In computer vision,vision,

extrapolating extrapolating such such plausible plausible content content outside outside the theframe frame boundaries boundaries is isknown as image known as image out-painting. out-painting.

[0002]

[0002] While imagesynthesis While image synthesismethods methods have have long long been been used used as aassolution a solution to to thisproblem, this problem,

more recently more recently learning-based learning-based methods methodswhich which leverage leverage learned learned priorsfor priors forthis this task task have have been shown been shown

to yield to yield more more promising results. For promising results. example, methods For example, methodshave have been been trained trained to to generateimages generate images that that

would likelyarise would likely ariseif ifoneone were were to continuously to continuously pan translate) pan (i.e., (i.e., translate) the camera. the camera.

SUMMARY SUMMARY

[00031

[0003] Introduced hereareare Introduced here techniques/technologies techniques/technologies that that generate generate a fulla 360-degree full 360-degree

panorama image from panorama image fromananinput inputnarrow narrowfield field of of view view image. image. Embodiments Embodimentsuse usea aguided guidedco- co

modulationgenerator modulation generatornetwork network to to enable enable users users to control to control thethe panorama panorama generation generation process. process. For For

example, example, aa pretrained pretrained guide guide model modelmay may support support various various classesofofscenes. classes scenes.The Theuser usercan canselect selectone one

of these these classes classesasasa guide which a guide whichis is then used then by by used thethe guided co-modulation guided co-modulationgenerator generatornetwork networkwhen when

generating the panorama. generating the panorama.AsAs such, such, rather rather than than a randomly a randomly predicted predicted panorama, panorama, the panorama the panorama

generation system generation system generates generates a varied a varied setresults set of of results consistent consistent with with the the image input inputand image and semantically semantically

matchingthethe matching desired desired class. class.

or more embodiments, camera parameters are estimated 2023200526 01 Feb 2023

[00041

[0004] More specifically, in More specifically, in one one or more embodiments, camera parameters are estimated

for for an input image an input imagetotogenerate generatea apanoramic panoramic projection. projection. For For example, example, the input the input imageimage may bemay be

warped basedononthethecamera warped based camera parameters parameters to form to form partpart of equirectangular of an an equirectangular representation representation of the of the

panorama. Thispanoramic panorama. This panoramic projectionisisthen projection thenprovided providedtotoa agenerator generator model modelwhich which generates generates a full a full

360-degree panorama 360-degree panorama using using the the image image and and a a guided guided user input. user input. A neural A neural network network is trained is trained to to

generate thepanoramic generate the panoramic projection projection to matching to have have matching edges, edges, this thisinresults results in no discontinuities no discontinuities across across

the full the full360-degree 360-degree panorama. panorama.

[0005]

[0005] Additional features Additional advantages andadvantages features and of exemplary of exemplary embodiments embodiments of the of the present present

disclosure willbebesetsetforth disclosure will forthin in thethe description description which which follows, follows, and in and part in part will will be from be obvious obvious the from the

description, description, or ormay may be learned by be learned by the practice practice of ofsuch such exemplary embodiments. exemplary embodiments.

BRIEF DESCRIPTION BRIEF DESCRIPTION OF OF THE THE DRAWINGS DRAWINGS

[0006]

[0006] The detailed description The detailed description is is described described with with reference reference to tothe theaccompanying drawings accompanying drawings

in in which: which:

[00071

[0007] FIG. diagram of illustrates a adiagram FIG. 11 illustrates ofaaprocess ofextracting processof 360-degree panoramas extracting360-degree from panoramas from

narrow field narrow field of view imagesin view images in accordance accordancewith withone oneorormore moreembodiments; embodiments;

[0008]

[0008] FIG. illustrates aa diagram FIG. 22 illustrates diagram of extracting 360-degree of aa pipeline extracting panoramas 360-degree panoramas from from

narrow field narrow field of of view imagesin view images in accordance accordancewith withone oneorormore moreembodiments; embodiments;

[0009]

[0009] illustrates a diagram FIG. 33 illustrates FIG. panorama from a 360-degreepanorama generatinga 360-degree a diagramofofgenerating from narrow field narrowfield

of view imagefor view image for in in accordance accordance with withone oneorormore moreembodiments; embodiments;

[0010]

[0010] FIG. FIG. 44 illustrates illustrates aa diagram diagramofofa amodel model architecture architecture for for extracting extracting 360-degree 360-degree

panoramasfrom panoramas from narrow narrow field field ofof view view images images forfor in in accordance accordance with with oneone or more or more embodiments; embodiments;

[0011]

[0011] FIG. FIG. 55illustrates illustrates aa diagram diagramof of a system a system for training for training a model a model to generate to generate 360-degree 360-degree

panoramasfrom panoramas from narrow narrow field field ofof view view images images forfor in in accordance accordance with with oneone or more or more embodiments; embodiments;

2 masksin in view masks of view field of accordance one one withwith or more 2023200526 01 Feb 2023

[00121

[0012] FIG. illustrates example FIG. 66 illustrates example field accordance or more

embodiments; embodiments;

[00131

[0013] example panorama illustrates example FIG. 77 illustrates FIG. panorama extrapolations in accordance extrapolations in accordance with oneor withone more or more

embodiments; embodiments;

[0014]

[0014] FIGS. 8-10illustrate FIGS. 8-10 illustrateananexample example panorama panorama extrapolation extrapolation and object and virtual virtual object

insertion insertion in inaccordance accordance with with one or more one or embodiments; more embodiments;

[0015]

[0015] FIG. FIG. 11 illustrates an 11 illustrates an example example of user interface of aa user accordance with in accordance interface in one or with one more or more

embodiments; embodiments;

[00161

[0016] schematic diagram illustrates a a schematic 12 illustrates FIG. 12 of panorama diagram of panorama generation systeminin generation system

accordance withone accordance with oneorormore moreembodiments. embodiments.

[00171

[0017] FIG. 13 illustrates FIG. 13 illustrates aa flowchart flowchart of of aa series series of of acts actsin ina amethod of extracting method of extracting 360 360-

degree panoramasfrom degree panoramas from narrow narrow fieldofofview field viewimages imagesininaccordance accordance with with oneone or or more more embodiments; embodiments;

and and

[00181

[0018] FIG. 14 illustrates a a block 14 illustrates block diagram of an diagram of an exemplary exemplarycomputing devicein in computingdevice

accordance withone accordance with oneorormore moreembodiments. embodiments.

DETAILED DESCRIPTION DETAILED DESCRIPTION

[0019]

[0019] One moreembodiments or more One or embodiments of the of the present present disclosure includea apanorama disclosureinclude panorama generation generation

system whichgenerates system which generatesananentire entire360-degree 360-degree view view around around the the camera camera basedbased on an on an input input image.image.

This effectively effectively extrapolates extrapolates the the field fieldof ofview view of of the thecamera camera to to span an entire entire sphere formed formedbyby

rotating the rotating the camera camera around around its its center center of projection. of projection. Virtual Virtual object object insertion insertion is a complex, is a complex, but critical but critical

component, of composition component, of 3D 3D composition and augmented and augmented reality. In reality. In particular, particular, it is especially it is especially challenging challenging to to

insert virtual objects insert virtual objectsinina avisually visually pleasing pleasing way, way, especially especially if the if the objects objects are are shiny or shiny or reflective. reflective.

This is is because, because, toto beberealistic realistic to to aa viewer, viewer, a ashiny shinyor or reflectiveobject reflective objectneeds needs to to reflect reflect thethe environment aroundthe thecamera. camera.However, However, a typicalimage image of of a ascene scenecaptures capturesonly onlya asmall smallfraction fraction 2023200526 01 Feb 2023 environment around a typical of the environment of the environment of that of that scene. scene.

[0020]

[0020] When a photograph When a photograph is taken, is taken, typically onlyonly the the typically information about about information the overall the overall

environment that environment that is is captured captured is the is the image image itself. itself. Accordingly, Accordingly, to fill to in fill thisinmissing this missing data, advanced data, advanced

techniques are required techniques are required to to hallucinate hallucinate the the entire entirelighting lightingenvironment from aa given environment from given image. image.This This

meansthat means that an an infinite infinite number number ofofpredicted predicted environment environment maps maps may may be generated be generated fromgiven from any any given

image. Prior techniques image. Prior techniques have used autoencoder have used autoencoder architectures architectures totoperform perform environment environment map map

extraction. extraction. Some of these Some of these techniques techniques may, may,for forexample, example,estimate estimatedirection directionlighting, lighting, ambient ambient low- low

frequency spherical harmonics, frequency spherical andan harmonics, and anHDR HDR cubemap. cubemap. However, However, this this technique technique typically typically produces produces

low resolution results. low resolution results. Other Othertechniques techniques require require thethe useruser to scan to scan the environment the environment (thereby (thereby

capturing additional environmental capturing additional environmentaldata) data) to to be used. be used. StillStill other other techniques techniques assume assume that the that the

environment mirrorsthe environment mirrors theimage, image,andand then then fillsininholes fills holesasasneeded neededusing using a generative a generative adversarial adversarial

network(GAN) network (GAN)or or patch-based patch-based algorithm. algorithm. This This cancan result result in in artifacts near artifacts near boundaries boundariesofofthe the input input

image as well image as well as as visually visually confusing imageswhen confusing images when looking looking at at a aglossy glossyorortransparent transparentobject, object, where where

objects areduplicated objects are duplicatedandand visible visible in front in front of, and of, and behind, behind, the camera the camera in the environment in the environment map. map.

[0021]

[0021] To addressthese To address and theseand other other deficienciesin in deficiencies conventional conventional systems, systems, embodiments embodiments

include a panorama include a generationsystem panorama generation system to to extrapolatea a360-degree extrapolate 360-degree panorama panorama fromfrom a narrow a narrow fieldfield

of view input image. view input image. An Animproved improvedGANGAN architecture architecture and training and training techniques techniques are used are used to convert to convert

an unconditional unconditional GAN GAN into into a conditional a conditional image image generator. generator. ThisThis allows allows for high for high resolution resolution and and

improved qualitypanoramas improved quality panoramas to generated to be be generated from from an an image, input input image, as compared as compared to previous to previous

techniques.Additionally, techniques. Additionally, the the GAN GAN is conditioned is conditioned on theimage on the input input andimage andinexecuted executed in a single a single feed- feed

forward pass, making forward pass, makingitit much muchfaster faster to to execute execute and and more moreaccurate accuratethan thantraditional traditional methods. methods.

4

[00221

[0022] In In some someembodiments, embodiments,camera parameters camera are estimated parameters for thefor are estimated input the image input to image to

generate a panoramic generate a projection. This panoramic projection. This panoramic panoramicprojection projectionisis then then provided providedto to aa generator generator model model

which generates which generatesa afull full 360-degree 360-degreepanorama panorama using using the the image image and and a a guided guided user input. user input. A neural A neural

networkis istrained network trained to to generate generate the the panoramic panoramic projection projection to have edges, to have matching matching this edges, results this in noresults in no

discontinuities acrossthethe discontinuities across full360-degree full 360-degree panorama. panorama.

[0023]

[0023] illustrates a adiagram FIG. 11 illustrates FIG. ofaaprocess diagram of ofextracting processof extracting 360-degree panoramas 360-degree from panoramas from

narrow field of view narrow field images in view images in accordance accordancewith withone oneorormore moreembodiments. embodiments. As shown As shown in FIG. in FIG. 1, a 1, a

panorama generationsystem panorama generation system 100100 receives receives an an input input 102102 which which comprises comprises an input an input imageimage 104 and 104 and

a guide 106, 106, at at numeral 1. The numeral 1. The input input image imagecan caninclude includeany anyimage image capturing capturing lessthan less thana full a full 360- 360

degree panorama.The degree panorama. The term term "image" "image" or "digital or "digital image" image" refers refers to to a digitalgraphics a digital graphicsfile file that that when when

rendereddisplays rendered displays oneone or more or more objects. objects. In particular, In particular, the"image" the term term "image" comprisescomprises a digital a digital file that, file that,

whenrendered, when rendered,includes includesvisual visualrepresentations representationsof of oneone or more or more objects, objects, suchsuch as a as a person. person. For For

example, example, thethe term term "digital "digital image" image" includes, includes, butlimited but is not is not to, limited to, files digital digital files with the with the following following

file fileextensions: extensions: JPG, JPG, TIFF, TIFF, BMP, PNG, BMP, PNG, RAW, RAW, or PDF. or PDF. The guide The guide 106 includes 106 includes a label a label that that is used is used

by the by the panorama generationsystem panorama generation system 100100 to to influence influence thethe panorama panorama thatthat is generated is generated forfor thethe input input

image. For example, image. For example,a aguide guideofof 'sky'generates 'sky' generatesa apanorama panorama thatthat emphasizes emphasizes the sky, the sky, a guide a guide of of

'lawn' adds 'lawn' adds grass grass to to the the generated generated panorama, etc. panorama, etc.

[0024]

[0024] numeral 2,2, a apanoramic At numeral At manager108108 projection manager panoramicprojection generates generates a panoramic a panoramic

projection of the projection the image. image. In In some embodiments,thethepanoramic some embodiments, panoramic projection projection manager manager 108 includes 108 includes a a

machinelearning machine learningmodel, model, such such as as a neural a neural network, network, which which has been has been trained trained to estimate to estimate cameracamera

parameters based parameters basedononthe theinput inputimage imageandand warp warp the the input input image image intointo a panoramic a panoramic projection. projection. For For

example, the input example, the input image imageis iswarped warped to to an an equirectangular equirectangular representation representation based based on a on a simplified simplified

camera modelusing camera model usingthe theestimated estimatedcamera camera parameters. parameters. A neural A neural network network may include may include a machine a machine- learning model that can can be be tuned tuned(e.g., (e.g., trained) trained) based based on on training training input input to toapproximate unknown 2023200526 01 Feb 2023 learning model that approximate unknown functions. In particular, functions. In particular, aaneural neuralnetwork can include network can include aa model modelofofinterconnected interconnecteddigital digital neurons neurons that communicate that and communicate and learnto toapproximate learn approximate complex complex functions functions and generate and generate outputs outputs based based on a on a plurality of plurality of inputs inputsprovided provided to to the themodel. model. For instance, the For instance, the neural neural network includes one network includes one or or more more machinelearning machine learningalgorithms. algorithms.In In otherwords, other words, a neural a neural network network is an is an algorithm algorithm thatthat implements implements deep learningtechniques, deep learning techniques, i.e., i.e., machine machine learning learning that utilizes that utilizes a set aof setalgorithms of algorithms to attempt to attempt to modelto model high-levelabstractions high-level abstractionsin in data. data.

[0025]

[0025] At numeral At thepanoramic numeral3,3,the panoramicprojection is is projection provided provided to panorama to panorama generator generator 110. 110.

Panorama generator110110 Panorama generator maymay include include a machine a machine learning learning model model trained trained to generate to generate a 360-degree a 360-degree

panorama fromananinput panorama from inputpanoramic panoramic projection.InInsome projection. some embodiments, embodiments, the the panorama panorama generator generator 110 110

is is aagenerator generatormodel model trained trainedasaspart partofof a generative adversarial a generative network adversarial (GAN). network Examples (GAN). Examples of of GANs GANs

include include StyleGAN, StyleGAN, CoModGAN, etc.Embodiments CoModGAN, etc. Embodimentsbuild buildon on CoModGAN CoModGAN by enabling by enabling guided guided co co-

modulation. InIn particular, modulation. particular, embodiments replacethe embodiments replace therandom random masks masks of CoModGAN of CoModGAN with with field of field of

view (FOV) view (FOV) masks. masks. Additionally, Additionally, embodiments embodiments add a horizontal add a horizontal shift shift to the to the generator generator output output

during training training before providing the generated providing the generated image imagetotothe thediscriminator discriminatormodel. model.This This trainsthe trains the

generator to generate generator to generate panoramas panoramasthat thatdodonot nothave have discontinuitiesatatthe discontinuities theedges. edges.Additionally, Additionally,the the

architectureisismodified architecture modifiedto to yield yield a 2:1 a 2:1 aspect aspect ratioratio to avoid to avoid anisotropic anisotropic upsampling upsampling artifacts artifacts when when

mappingthe mapping theoutput outputtoto the the equirectangular equirectangular representation. representation.

[0026]

[0026] At numeral4, At numeral the panorama 4, the generatorgenerates panorama generator panorama extrapolated panorama generatesananextrapolated from from thethe

input image andthe image and the guide. guide. The Theresulting resulting extrapolated extrapolated panorama panorama120120 is is then then output output at atnumeral 5. 5. numeral

The extrapolated The extrapolated panorama panoramamaymay be be used used in variety in variety of of applications.For applications. Forexample, example, in in virtualobject virtual object

compositing anobject compositing an objectisis added addedtoto ananimage. image.When Whenthe the object object hashas a shiny a shiny or or otherwise otherwise reflective reflective

surface, surface, one wouldexpect one would expect it it totoreflect reflectportions portionsofof theenvironment the environment in which in which the image the image was was captured. However,when when presented with with an image, there is no data available for what for the what the 2023200526 01 Feb 2023 captured. However, presented an image, there is no data available surroundings mayplausibly surroundings may plausiblylook looklike. like. By Byfirst first extrapolating extrapolating aapanorama from the panorama from the input input image, image, the the resulting extrapolated resulting extrapolated panorama panorama is to is used used addtoreflections add reflections to the composited to the composited objects, in objects, resulting resulting a in a morerealistic more realisticand and visually visually pleasing pleasing scene. scene.

[00271

[0027] FIG. illustrates aa diagram FIG. 22 illustrates of aa pipeline diagram of extracting 360-degree pipeline extracting panoramas 360-degree panoramas from from

narrow field narrow field of of view imagesininaccordance view images accordancewith with oneone or or more more embodiments. embodiments. As shown As shown in FIG.in2,FIG. 2,

an input image an input image200 200isisprovided providedto topanorama panorama generation generation system system 100.this 100. In In this example, example, the input the input

image depictsa abuilding, image depicts building,parking parking lot,trees lot, treesandand sky. sky. The The panorama panorama generation generation system system then then

generates generates a a panoramic projection 202 panoramic projection 202ofofthe the input input image. As shown, image. As shown,the thepanoramic panoramicprojection projection202 202

has has been warped based been warped based on onestimated estimated camera camera parameters. parameters. In In some some embodiments, embodiments,the thecamera camera

parameters are parameters are inferred inferred using using aa camera cameraparameter parameterestimation estimation model model 206.206. Alternatively, Alternatively, different different

techniques may techniques maybe be used used to generate to generate the the panoramic panoramic projections projections 202 and/or 202 and/or obtain obtain the the camera camera

parameters. For parameters. For example, example, inin some someembodiments, embodiments,thethe camera camera parameters parameters may may be known be known (e.g.,(e.g., basedbased

on on aa known known camera camera model, model, included included as metadata as metadata with with the the image, input input etc.). image, The etc.). The panoramic panoramic

projection 202 projection 202 then thenserves servesasasinput inputtotothe thepanorama panorama generator generator (e.g., (e.g., generative generative model model 208 208 as as

shown) which shown) which then then generates generates a panorama a panorama 204 the 204 from from the panoramic panoramic projection. projection. Asthis As shown, shown, this

generates a full full360-degree 360-degree panorama. panorama.

[0028]

[0028] FIG. illustrates a diagram FIG. 33 illustrates panorama from a 360-degreepanorama generatinga 360-degree a diagramofofgenerating from narrow field narrowfield

of view imagefor view image forininaccordance accordancewith with oneone or or more more embodiments. embodiments. As in As shown shown FIG. in 3, FIG. 3, in some in some

embodiments, thepanorama embodiments, the panorama generation generation system system 100 100 includes includes the the panorama panorama generator generator but not but does does not

include components componentsforforgenerating generatingthethepanoramic panoramic projection. projection. TheThe panoramic panoramic projection projection may be may be

obtained from obtained from a separate a separate system system or service, or service, generated generated by the by the user, user, etc. etc.

of a of architecture a panorama system forsystem for generation 2023200526 01 Feb 2023

[00291

[0029] FIG. diagram illustrates aadiagram FIG. 44illustrates of an of an architecture panorama generation

extracting extracting 360-degree panoramasfrom 360-degree panoramas from narrow narrow field field of of view view images images for for in in accordance accordance withwith one one or or

more embodiments. more embodiments.As As shown shown in FIG. in FIG. 4, architecture400400 4, architecture includes includes a guidemodel a guide model 402, 402, a panoramic a panoramic

projection manager projection manager404, 404,andand panorama panorama generator generator 406. 406. As discussed, As discussed, the panorama the panorama generation generation

system receives an system receives an input input image image408 408which whichhashas a lessthan a less than360-degree 360-degreefield fieldofofview, view,and andoutputs outputsa a

360-degree panorama 360-degree panorama 410. 410.

[0030]

[0030] As discussed, As discussed, the the input input image image408 408is isprovided provided to to panoramic panoramic projection projection manager manager

404 to 404 to generate generate panoramic panoramicprojection projection412. 412.AsAs discussed, discussed, embodiments embodiments extrapolate extrapolate a 360-degree a 360-degree

field field of of view view from aninput from an input image. image.This Thiscan canbebeframed framed as an as an out-painting out-painting in an in an equirectangular equirectangular

panoramicrepresentation. panoramic representation. AsAsa afirst first step, step, the the input input image 408 is image 408 is warped warpedinto into ananequirectangular equirectangular

representation representation 412: 412:

pi m == KRpw Pim KRpw

[0031]

[0031] Here, Here, Pw represents aa 3D pwrepresents pointinin world 3Dpoint worldcoordinates (whose coordinates(whose origincoincides origin with coincideswith

the center the centerofofprojection projectionof of thethe camera, camera, hencehence no translation), no translation), and Pim and pim represents represents its projection its projection in in

homogeneous homogeneous coordinates coordinates on on thethe image image plane. plane. A pinhole A pinhole camera camera model model with with common common assumptions assumptions

(e.g., (e.g., principal pointisisthe principal point theimage image center, center, negligible negligible imageimage skew, skew, unit unitaspect pixel pixelratio, aspectetc.), ratio,canetc.), can

be used be to yield used to yield KK == diag([f diag([f f 1]),where f 1]), where f f is isthe thefocal focallength length in in pixels. pixels. In In some embodiments, some embodiments,

h in in the therepresentation f is freplaced representation with the is replaced vertical with field of view the vertical he of field = 2arctan(-), where view h = where 2f) h isthe h is the

image height. image height.

[0032]

[0032] The The rotation matrix R rotation matrix can be R can roll, y, parameterizedbybyroll, be parameterized pitch P, , pitch and yaw ß, and Sinceanan yaw .a.Since

arbitrary imagepossesses arbitrary image possesses no natural no natural reference reference frame frame to to estimate estimate a, ittois0,set , it is set andto is 0, and also yset is to also set to

0. 0. This This results resultsininR R== R,(p). In some Rx(). In someembodiments, embodiments, camera camera parametersf parameters f and and Pare ß are assumed assumed to beto be

8 known.Alternatively, Alternatively, the the camera cameraparameters parametersmaymay 2023200526 01 Feb 2023 known. be be inferred inferred or or estimated estimated from from the the input input image, image, such as using a camera such as estimationmodel, parameterestimation camera parameter model,asasdiscussed above. discussedabove.

[0033]

[0033] As shown As shownininFIG. FIG.4,4,panorama panorama generator generator 406406 provides provides a co-modulated a co-modulated GAN, GAN, such such

as as the the CoModGAN architecture. CoModGAN architecture. A co-modulated A co-modulated GAN receives GAN receives a warped a warped input and input image image thenand an then an

encoder 414co-modulates encoder 414 co-modulates synthesis synthesis network network 416 with 416 with mapper mapper 418 onbased 418 based on theinput the warped warped input

image 412. Such image 412. Sucha a co-modulated co-modulated GAN GAN architecture architecture alonealone produces produces high quality high quality results results for for image image

completion butisis not completion but not suitable suitable for for full full panorama extrapolation. For panorama extrapolation. Forexample, example,random random masks masks are are

usedduring used duringtraining training which which do represent do not not represent the typical the typical scenarios scenarios of FOV of FOV extrapolation. extrapolation. This This results results

in blurry results results at at test testtime. time.Additionally, Additionally,since sincesuch such networks are designed networks are designedfor forinpainting inpaintinginin

perspectiveimages, perspective images, distinct distinct artifacts artifacts are are produced produced along along theofedge the edge of the panorama the panorama (e.g., (e.g., where the where the

image "wraps-around"). These image "wraps-around"). shortcomings are These shortcomings are addressed addressed below below via via aa number of improved number of improved

training techniques. training techniques.

[0034]

[0034] Additionally, as discussed, Additionally, as discussed, prior techniquescancan prior techniques predict an an predict infinite infinite number of of number

environment maps, environment maps, making making it difficult it difficult to identify to identify appropriate appropriate maps formaps for any any given given application. application. This This

lack lack of of control controlover overoutput outputpresents presentsa a major majorchallenge challengetoto implementing implementing such such systems. systems. Accordingly, Accordingly,

embodiments embodiments useuse another, another, pre-trained pre-trained network, network, in concert in concert with with theasGAN the GAN as for a guide a guide co- for co

modulation.TheThe modulation. guide guide modelmodel 402 is 402 is a trained a trained model model that that apredicts predicts a class associated class associated with with the input the input

image. The image. The class class describes describes the characteristics the characteristics of theof the image, image, such as such as picnic picnic area, sky, area, cliff,sky, street, cliff, street,

lawn, etc. etc. Various Various classification classification models modelsmay may be be usedused as guide as guide modelmodel 402. 402. The Themodel guide guide model

produces aa latent produces latent vector vector gg 420 420 from the input from the input image 408. image 408.

[0035]

[0035] In In some embodiments, some embodiments, thethe vectorg g420 latentvector latent 420that to to corresponds thatcorresponds theclass the which classwhich

is is provided to the provided to the panorama panoramagenerator generator406. 406. If If theuser the userprovides provides a differentinput a different inputguide, guide,then then a a

difference betweenthe difference between thedetermined determinedtarget targetclass classt tofofthe the input input image imageand and theuser-selected the user-selectedclass, class,

9 represented as as t*, t*, isis calculated calculated and and passed backthrough throughthe theguide guidemodel model 402 402 to determine the 2023200526 01 Feb 2023 represented passed back to determine the corresponding latent vector corresponding latent vectorg*g*(e.g., (e.g., aa target vector) which target vector) whichisisthen thenprovided provided to to thethe panorama panorama generator 406. In generator 406. In some someembodiments, embodiments, both both g and g and g* are g* are provided provided to panorama to the the panorama generator. generator. In In some embodiments, some embodiments, onlyonly g org g* orisg*provided. is provided. This latent This latent vectorvector g 420 gis420 thenisprovided then provided to a to a classification classificationsubnetwork 422 tt == c(g). subnetwork 422 c(g).Here Heret tE ER~RN is is thevector the vectorofofpredicted predictedprobabilities probabilities over over

N classes. N classes. The guided co-modulated The guided co-modulatedvector vectorw'w'424424 is is

w' = w' = AA(G(x),M(z)) (g (x), M (z))

[0036]

[0036] In In the the above equation, G(x) above equation, g(x) is is the the output output of of the the guided modelapplied guided model appliedtoto the the input input

image, x, M(z) image, x, M(z)is is thethe output output of mapper of mapper 418 applied 418 applied to z, to input and z, input anda A A is is a learnable learnable affine affine

transformer that transformer that combines combinesthetheoutput outputof of thethe encoder, encoder, thethe mapper, mapper, and and the guide the guide network. network. The The

guided co-modulatedoutput guided co-modulated outputandand thethe output output of of thethe encoder encoder 414414 is then is then provided provided to the to the synthesis synthesis

network416 network 416ofofpanorama panorama generator generator 406406 which which generates generates the the fullfull 360-degree 360-degree panorama panorama 410. 410.

[00371

[0037] Embodiments cantune Embodiments can tunethe the output output appearance appearance of of the the output output panorama panorama 410 410byby

modifying the latent vector g to represent another class. This can be done by optimizing a one-hot modifying the latent vector g to represent another class. This can be done by optimizing a one-hot

vector vector ti with withthe thedesired desired class class

g* g* == argmin argmin-e(c(g),£) l(c(g), t) 9 g

[0038]

[0038] is aa binary where l-eis where binarycross-entropy function. AApanorama, lossfunction. cross-entropyloss panorama, whose whose appearance appearance

outside outside the the FOV FOV ofofthe the input input image imageshould shouldbetter better match matcht iisis produced producedbybyreplacing replacingggg*, <- i.e., g*, i.e.,

w' = w' = A (g *, M (z)) A(g*,M(z))

[0039]

[0039] In In contrast contrast to to existing existing editing editing methods, embodimentsdo do methods, embodiments notnot require require training training thethe

guide modelononthethedomain guide model domain output output by synthesizer by the the synthesizer (panoramas), (panoramas), any pre-trained any model model pre-trained on on

regular images regular images cancan do. do. It also It also doesdoes not require not require any analysis any analysis of the learned of the learned latent space. latent space.

10

FIG. 5 illustrates a diagram of a system for training a model to generate 360-degree

[00401

[0040] FIG. 5 illustrates a diagram of a system for training a model to generate 360-degree

panoramasfrom panoramas from narrow narrow field field of of view view images images for for in accordance in accordance withwith onemore one or or more embodiments. embodiments.

Training thepanorama Training the panorama generator generator startsstarts with obtaining with obtaining a training a training dataset dataset that that includes includes a plurality a plurality of of

full full panorama images500. panorama images 500.The Thefull fullpanorama panorama images images may may be obtained be obtained from from any available any available source source

of 360-degree of 360-degree panoramas panoramasand and do need do not not need to be to be labeled. labeled. When training When training (and for(and for validation), validation),

randomcrops random cropsare arecomputed computedon on thethe flyflytotoensure ensureasasdiverse diverseaa set set as as possible. possible. For Forexample, example, in in some some

embodiments, theparameters embodiments, the parametersarearesampled sampled as as h ~hoU(40,120) - U(40,120) for the for the FOV FOV and ß and P - N(0,30) ~ N(0,30) for the for the

elevation angle, angle, where U and where U and NNare are uniform uniformand andnormal normaldistributions distributions respectively. respectively. After After sampling, sampling,

P ß is is clipped clippedtoto[-30,30]°. A Atesting

[-30,30]°. setset testing of panoramas of panoramasmay maybe bebalanced balanced between outdoorand between outdoor andindoor indoor

scenes thatare scenes that arenot notused used during during the the training. training.

[0041]

[0041] As FIG. shownin inFIG. As shown 5, from the the 5, from fullfull panoramas 500, 500, panoramas various various FOV FOV masks masks

corresponding corresponding totodifferent different camera cameraparameters, parameters,elevations, elevations,etc. etc.are areapplied applied502 502to toobtain obtainpartial partial

panoramas panoramas 506. 506. Instead Instead ofrandom of the the random masking masking of prior techniques, of prior techniques, this resultsthis in aresults set of in a set training of training

images similar to images similar to what what will will be be obtained obtainedatat test test time time from an input from an input image. image. These Theseimages images areare then then

provided to provided to panorama panoramagenerator generator 508, 508, which which includes includes an encoder an encoder 510, 510, a mapper a mapper 512, a512, a learnable learnable

affine affine transform 514and transform 514 anda asynthesis synthesisnetwork network 516.516. OnceOnce trained, trained, the panorama the panorama generator generator 508 508

results ininthe results thepanorama panorama generators described described above. above.

[0042]

[0042] The output The outputofofthe thepanorama panorama generator generator 508 508 is generated is generated panoramas panoramas 518. During 518. During

training these training these are areprovided provided to to aa discriminator discriminatornetwork 520 which network 520 whichcompares comparesthethe generated generated images images

to the to the full fullpanoramas 500 and panoramas 500 anddetermines determineswhether whether thethe generated generated image image is real is real or or fake.Based fake. Based on on

the result, the result, the discriminator network the discriminator network520 520 andand panorama panorama generator generator 508 are508 thenare then trained. trained. As As

discussed,traditional discussed, traditionaltraining training techniques techniques lead lead to visual to visual artifacts artifacts wherewhere the panorama the panorama wraps wraps around around

on itself. To on itself. To address address this this issue, issue,atatleast some least someof ofthe thefull fullpanoramas panoramas are are rolled rolled 522 522 before before being being

11 providedtotothethediscriminator discriminator network. Thisrepresents roll represents a horizontal shiftpanorama of the and panorama and 2023200526 01 Feb 2023 provided network. This roll a horizontal shift of the encourages the generator encourages the generatortotolearn learn to to produce producepanoramas panoramas with with no discontinuities no discontinuities at at thethe edges.In In edges.

some embodiments, some embodiments, the operator the roll 522 is 522 roll operator is additionally, additionally, or alternatively, or alternatively, applied toapplied to the the output of output of

the panorama the generatorbefore panorama generator beforebeing beingprovided providedtotothe thediscriminator. discriminator.

[0043]

[0043] FIG. illustrates example FIG. 66 illustrates field of example field masksin in view masks of view accordance accordance one one withwith or more or more

embodiments. As discussed, embodiments. As discussed, prior priorco-modulated co-modulatedGANs were trained GANs were trained using using random masks. An random masks. An

example ofsuch example of suchrandom randommasks masks is is shown shown at 600. at 600. These These masks masks would would be applied be applied to images to images to create to create

training images training that were images that wererandomly randomly masked. masked. This This wouldwould lead lead to to training training a generator a generator that that has has

learned toin-paint learned to in-painta aplausible plausible missing missing content content under under a wide avariety wide variety of circumstances. of circumstances.

[0044]

[0044] However, suchtraining However, such doesnotnotresemble trainingdoes thethe resemble problem problem of in-painting of in-painting panoramas panoramas

from an input from an input narrow narrowFOV FOV image. image. Accordingly, Accordingly, in some in some embodiments, embodiments, the guided the guided co-modulated co-modulated

GAN GAN is istrained trainedusing usingFOV FOV masks masks 602.602. Although Although six example six example FOVare FOV masks masks shownare at shown at 602, this 602, this

is is intended to be exemplary, intended to exemplary, and andnot notlimiting, limiting, asasa avariety variety ofofFOV FOV masks masks may may be be during used used during

training. As training. As discussed above, full discussed above, full panoramas areused panoramas are usedduring duringtraining. training. These Thesefull fullpanoramas panoramasareare

combined withFOV combined with FOV masks masks 602create 602 to to create the partial the partial panoramas panoramas described described above. above. TheseThese resemble resemble

the the warped narrowFOV warped narrow FOV images images thatthat are are obtained obtained at test at test time,providing time, providing a more a more accurate accurate training training

set for the set for network. the network.

[0045]

[0045] example panorama illustrates example FIG. 77 illustrates FIG. panorama extrapolations in accordance extrapolations in withone accordance with or more oneor more

embodiments. embodiments. AsAs shown shown in in FIG. FIG. 7, 7, thetheuser-provided user-provided guide guide leadstotodifferent leads different generated generated panoramas panoramas

based on based on the the guide. guide. For For example, the input example, the input image remainsthe image remains the same sameinin each each extrapolated extrapolated panorama panorama

(corresponding to the (corresponding to the center center of each each image). However, theextrapolated However, the extrapolatedportions portionsofofthe the panorama panorama

vary, vary, depending onthe depending on theselected selected guide. guide. This Thisresults results in in more controllable panorama more controllable panoramageneration generation as as

compared compared totoprevious previoustechniques. techniques.

12

[00461 FIGS. 8-10illustrate illustrateananexample Feb 2023

[0046] FIGS. 8-10 example panorama panorama extrapolation extrapolation and object and virtual virtual object

insertion insertion in in accordance with one accordance with oneorormore more embodiments. embodiments. FIG. FIG. 8 shows 8 shows an example an example of an input of an input

image 800. Although image 800. AlthoughFIG. FIG.8 illustrates 8 illustrates an an example exampleofofananimage imageofofananinterior interior scene, scene, embodiments embodiments 2023200526 01

can be used can be used on on images imagesrepresenting representingexterior exterior and/or and/or interior interior scenes. scenes. As As discussed, discussed, the the input input image image

is is then then used, used, along along with with aa guide, guide, to tocreate createananextrapolated extrapolatedpanorama panorama image, an example image, an exampleofofwhich which

is is shown in FIG. shown in 9. As FIG. 9. As shown in the shown in the example ofFIG. example of FIG.9,9, the the extrapolated panorama image900900 panorama image now now

includes aa full full 360-degree viewofofthe 360-degree view theinterior interior scene, scene, with withadditional additionaldetails details generated generatedfor forthe the

hallway, windows hallway, windowsonon thewalls, the walls,etc. etc. Once Oncecompleted, completed,thetheextrapolated extrapolatedpanorama panorama can can be used be used in ain a

variety variety of of ways. ways. For example, example, the the panorama panoramacancan be be added added to to a panorama a panorama viewer viewer whichwhich enables enables a a

user to user to look look in in any any direction direction and andsee seea adifferent differentportion portionofofthe thescene scene (such (such as as in in a virtual a virtual or or

augmented reality environment). augmented reality environment).

[00471

[0047] Additionally, Additionally, in example ofofFIG. the example in the virtualobjects FIG.10,10,virtual canbebeinserted objectscan insertedinto the intothe

scene. Asdiscussed, scene. As discussed, shiny shiny objects objects present present a particular a particular challenge challenge when because when inserted insertedthere because is no there is no

environmental datafor environmental data for them themtoto reflect, reflect, making themappear making them appearunrealistic unrealistic and/or and/or visually visually confusing. confusing.

However,asasshown However, shownin in theexample the example of of FIG. FIG. 10,10, because because a complete a complete panorama panorama has been has been generated, generated,

inserted whichreflect inserted which reflect the theappropriate appropriatecontent content from from the extrapolated the extrapolated panorama, panorama, making making the the

presence of presence of the the objects objects more realistic. For more realistic. Forexample, example, in in augmented image1000, augmented image 1000,three threenew new objects objects

1002 have been 1002 have beenadded. added.The Themirrored mirrored spheres spheres reflectthe reflect thelight light from from the the environment, environment,making making areas areas

outside the the FOV FOVof of thethe originalimage original image visible.Because visible. Because thatthat areaarea has has beenbeen generated generated (e.g.,(e.g., in in

extrapolated panorama900), extrapolated panorama 900),the theextrapolated extrapolatedpanorama panorama data data cancan be used be used to render to render the the reflection reflection

shown shown ininthe the spheres. spheres. This This makes makesthetheinsertion insertionofofthe thespheres spheresmore morerealistic/visually realistic/visually pleasing pleasing as as

they appeartotoplausibly they appear plausibly exist exist within within the space the space represented represented by the image. by the image.

13 of aa user user interface in accordance interface in one or with one more or more 2023200526 01 Feb 2023

[00481

[0048] FIG. illustrates an 11 illustrates FIG. 11 an example example of accordance with

embodiments. The embodiments. The example example useruser interface interface 11001100 of FIG. of FIG. 11 shows 11 shows one implementation one implementation of a user of a user

interface, interface, various various different differentuser userinterface interfaceimplementations implementations may also be may also be used usedininaccordance accordancewith with

various embodiments. The embodiments. The user user provides provides anan inputimage input image 1102 1102 via via thethe user user interface1100. interface 1100.InInsome some

embodiments, theuser embodiments, the useruploads uploadsthe theimage, image,downloads downloadsthethe image, image, provides provides a link a link oror otheridentifier other identifier

associated with with the the image imagestored storedlocally locally or or remotely, remotely, etc. etc. The Theuser usermay mayalso alsooptionally optionallyselect selecta a

guide 1103 from guide 1103 froma alist list of of guides guides supported supported by by the the guide guide model. For example, model. For example, the the supported supportedguides guides

mayinclude may include thethe classes classes which which the guide the guide model model is is to trained trained to identify. identify.

[0049]

[0049] The The user can then user can select aa user then select interface element user interface 1104totogenerate element 1104 generatethe panoramic thepanoramic

projection 1106. projection 1106. The The panoramic panoramicprojection projectionisishen hengenerated generatedasasdiscussed discussedabove. above.For Forexample, example, thethe

camera parametersare camera parameters aredetermined determined (e.g., estimated, (e.g., estimated, obtained obtainedfrom fromimage image metadata, metadata, etc.)and etc.) andused used

to to warp the input warp the inputimage imageto tocreate createthethepanoramic panoramic projection. projection. The The user user can the can edit edit panoramic the panoramic

projection1106 projection 1106 as needed as needed (e.g.,(e.g., can adjust can adjust theoffield the field view,of view, elevation, elevation, etc.). The etc.). Thethen user can user can then

select select aa user user interface interfaceelement element 1108 1108 to to extrapolate extrapolatethe thepanorama. panorama. The The panorama is then panorama is then

extrapolated as as discussed discussed herein. herein. For example, using For example, usingthe theselected selected guide, guide, the the panorama panoramagenerator generator

performs guided performs guidedco-modulation co-modulation generation generation to create to create the complete the complete panorama. panorama. The The resulting resulting

panoramaisisthen panorama thenpresented presented in panorama in a a panorama viewerviewer 1110 enables 1110 which which the enables the view user to userthe to view the

panorama from panorama from a varietyofofangles. a variety angles.Once Once satisfied,the satisfied, the user user may maydownload download the the panorama panorama for use for use

in other applications in other applications(e.g., (e.g.,object object insertion, insertion, etc.). etc.).

[0050]

[0050] FIG. 12illustrates FIG. 12 illustrates aa schematic schematicdiagram diagram of panorama of panorama generation generation system system (e.g., (e.g.,

"panorama generation"described "panorama generation" describedabove) above) inin accordance accordance with with oneone or or more more embodiments. embodiments. As shown, As shown,

the panorama the generationsystem panorama generation system 1200 1200 may may include, include, but but is not is not limited limited to, to, user user interfacemanager interface manager

1202, panorama projectionmanager panorama projection manager 1204, 1204, neural neural network network manager manager 1206,1206, training training manager manager 1208, 1208,

14 and storage manager manager1210. 1210. The The neural network manager 1204 includes generator generator a panorama 2023200526 01 Feb 2023 and storage neural network manager 1204 includes a panorama

1212, aa guide model 1214,and model 1214, anda adiscriminator network1216. discriminator network 1216.The The storage storage manager manager 12101210 includes includes

input input image data 1218, image data 1218, panoramic panoramicprojection projectiondata data1220, 1220,generated generated panoramas panoramas 1222,1222, and training and training

data data 1224. 1224.

[0051]

[0051] As in FIG. illustrated in As illustrated the panorama 12, the FIG. 12, generationsystem panoramageneration 12001200 system includes includes a user a user

interface interface manager 1202.For manager 1202. Forexample, example, thethe user user interfacemanager interface manager 1202 1202 allows allows usersusers to provide to provide

input image data to image data to the thepanorama generation system panorama generation system 1200. 1200. In In some someembodiments, embodiments,thethe user user interface interface

manager1202 manager 1202provides provides a userinterface a user interfacethrough throughwhich which thethe usercan user canupload upload thethe inputimages input images 1218 1218

for for which which aa panorama panoramais istotobebegenerated, generated,asasdiscussed discussedabove. above. Alternatively,ororadditionally, Alternatively, additionally,the the

user interface interface may may enable the the user to to download the images download the imagesfrom froma alocal local or or remote remote storage storage location location

(e.g., (e.g., by by providing providing anan address address (e.g., (e.g., a URL a URL or other or other endpoint) endpoint) associated associated with source). with an image an imageIn source). In

some embodiments, some embodiments, thethe user user interfacecan interface canenable enablea auser usertoto link link an an image imagecapture capture device, device, such such as as aa

camera or other camera or other hardware hardwaretotocapture capture image imagedata dataand andprovide provideititto to the the panorama generationsystem panorama generation system

1200. 1200.

[0052]

[0052] the user Additionally, the Additionally, userinterface manager 1202 interfacemanager 1202 allows users to allows users request the to request thepanorama panorama

generation system1200 generation system 1200to togenerate generate an an extrapolated extrapolated panorama panorama for input for the the input imageimage and select and select a a

guide guide for for the thepanorama generation. In panorama generation. In some embodiments,thetheuser some embodiments, userinterface interface manager manager1202 1202 enables enables

the user user to to view view the the resulting resulting generated generated panorama and/orperform panorama and/or performother otheractions actionsonon thepanorama the panorama

(e.g., (e.g., object insertion, etc.). object insertion, etc.).

[0053]

[0053] As illustrated in As illustrated in FIG. 12, the FIG. 12, panoramageneration the panorama system generationsystem also also 12001200 includes includes a a

panoramicprojection panoramic projectionmanager manager 1204. 1204. As As discussed, discussed, thethe panorama panorama projection projection manager manager can receive can receive

the input the input image image1218 1218 andand determine determine camera camera parameters parameters associated associated with thewith theInimage. image. some In some

embodiments, this may embodiments, this mayinclude include estimating estimating camera cameraparameters parametersusing usinga acamera camera parameters parameters

Feb 2023 estimation model. model. Alternatively, Alternatively, the the camera cameraparameters parametersmay maybe be known known (e.g., (e.g., thethe image image is from is from a a

camera withknown camera with known parameters, parameters, thethe camera camera parameters parameters are are associated associated with with thethe image image as metadata, as metadata,

etc.). etc.).Using Using the thecamera camera parameters, parameters, the the panorama projection manager panorama projection manager1204 1204 warps warps the the input input image image

2023200526 01

to to create create panoramic projection 1220. panoramic projection 1220.

[0054]

[0054] in FIG. illustrated in As illustrated As the panorama 12, the FIG. 12, generation panoramageneration system 12001200 system also also includes includes a a

neural network neural networkmanager manager 1206. 1206. Neural Neural network network manager manager 1208 may1208 host may host a of a plurality plurality neural of neural

networks or networks or other other machine machinelearning learningmodels, models,such suchasaspanorama panorama generator generator 1212, 1212, guide guide model model 1214, 1214,

discriminator network1216, discriminator network 1216,etc. etc.The Theneural neural network network manager manager 1206include 1206 may may include an execution an execution

environment, libraries, and/or environment, libraries, and/or any other data any other data needed to execute needed to execute the the machine machinelearning learningmodels. models. In In

some embodiments, some embodiments, thethe neural neural network network manager manager 1206 1206 may may be be associated associated with dedicated with dedicated software software

and/or hardwareresources and/or hardware resourcestotoexecute executethe themachine machine learning learning models. models. As discussed, As discussed, the the panorama panorama

generator 1212may generator 1212 may include include a guided a guided co-modulation co-modulation generator generator which receives which receives the panoramic the panoramic

projection 1220 projection 1220 and and aa guide guide value value and andgenerates generatesgenerated generatedpanorama panorama 1222. 1222.

[0055]

[0055] The neural network The neural networkmanager manager 1206 1206 may may alsoalso include include guide guide model model 1214 which 1214 which may may

be a pretrained be a pretrained classifier, classifier, as as discussed above. discussed InIn above. some someembodiments, the neural embodiments, the neural network manager network manager

1206 mayalso 1206 may alsohost host discriminator discriminator 1216 1216 which which is used is used to train to train the panorama the panorama generator generator 1212. 1212.

Although depictedinin FIG. Although depicted FIG. 1212as as being being hosted hosted by by aa single single neural neural network manager1206, network manager 1206,ininvarious various

embodiments theneural embodiments the neuralnetworks networks maymay be hosted be hosted in multiple in multiple neural neural network network managers managers and/or and/or as as

part of different part different components. Forexample, components. For example,each each model model 1212-1216 1212-1216 can becan be hosted hosted by own by their their own

neural networkmanager, neural network manager,or or other other host host environment, environment, in which in which the respective the respective neuralneural networks networks

execute, execute, or or the themodels models may be spread may be spread across across multiple multiple neural neural network managersdepending network managers depending on,on, e.g., e.g.,

the resource the resource requirements of each requirements of each model, model,etc. etc.

16 panorama generation system also includes 1200includes 2023200526 01 Feb 2023

[00561

[0056] As illustrated As in FIG. illustrated in thepanorama FIG. 1212the generation system 1200 also

training manager training 1210. The manager 1210. Thetraining training manager manager1210 1210cancan teach,guide, teach, guide,tune, tune, and/or and/ortrain train one one or or more more

neural networks. InInparticular, neural networks. particular, the the training training manager 1210can manager 1210 cantrain trainaa neural neural network networkbased based on on a a

plurality of plurality of training training data data 1224. For example, 1224. For example,the thepanorama panorama generator generator 1212 1212 may may be be trained trained to to

generate aa panorama from panorama from an an input input narrow narrow FOV FOV image. image. Additionally, Additionally, the may the GANs GANs may be further be further

optimized usingloss optimized using loss functions, functions, as as discussed above, by discussed above, bybackpropagating backpropagatinggradient gradient descents.More descents. More

specifically, specifically, the the training trainingmanager 1210 can manager 1210 canaccess, access,identify, identify, generate, generate, create, create, and/or and/or determine determine

training input training inputand and utilizethethe utilize training training input input to train to train and and fine-tune fine-tune a neural a neural network. For instance, network. For instance,

the training the training manager 1210cancan manager 1210 trainthethepanorama train panorama generator generator 1212 1212 usingusing discriminator discriminator network network

1216, end-to-end, as 1216, end-to-end, as discussed above. discussed above.

[00571

[0057] As illustrated in As illustrated FIG. 12, in FIG. the panorama 12, the panorama generation 1200 system1200 generationsystem also also includes thethe includes

storage manager1210. storage manager 1210.TheThe storage storage manager manager 1210 maintains 1210 maintains data data for thefor the panorama panorama generation generation

system 1200. The system 1200. Thestorage storagemanager manager 12101210 can can maintain maintain datadata of any of any type, type, size, size, or or kind kind as as necessary necessary

to to perform the functions perform the functions ofofthe thepanorama panorama generation generation system system 1200.1200. As in As shown shown in FIG. FIG. 12, the 12, the

storage storage manager 1210includes manager 1210 includesinput inputimage imagedata data1218. 1218.Input Input image image data data 1218 1218 can can include include data data forfor

any digital image any digital utilized by image utilized the panorama by the generationsystem panorama generation system 1200. 1200. For example, For example, input input image image

data data 1218 includes aa narrow 1218 includes FOVdigital narrow FOV digital image imageofofaascene scene provided providedby byaauser, user, where the user where the user wants wants

to have to have aa full full 360-degree panorama 360-degree panorama generated generated from from the the image. image. In context, In this this context, a narrow a narrow FOV FOV

image can image can include include any any imageimage with awith fieldaof field viewof view less less than than360a full a full 360 degrees. degrees.

[0058]

[0058] The storage The manager1210 storage manager maymay 1210 alsoalso include include panoramic panoramic projection data data projection 1220.1220. The The

panoramicprojection panoramic projection data data 1220 1220may mayinclude includea awarped warped inputimage input image thathas that hasbeen been warped warped according according

to camera to parameters associated camera parameters associated with with the the input input image. image. The panoramicprojection The panoramic projectiondata data1220 1220reflects reflects

the portion the portion of of the the complete complete 360-degree 360-degree FOV FOV represented represented by by the the input input image. image. In some In some embodiments, embodiments, thethe panoramic projection data data 1220 is stored by the by the storage manager during 2023200526 01 Feb 2023 panoramic projection 1220 is stored storage manager during processing oror totobebeused processing used to to generate generate additional additional panorama panorama options options at a time. at a later later In time. someIn some embodiments, thepanoramic embodiments, the panoramic projection projection data data 1220 1220 is is maintained maintained in in memory memory during during processing. processing.

[0059]

[0059] The storage manager The storage 1210 manager1210 may may further further include include generated generated panorama datadata panorama 1222. The The 1222.

generated panorama generated panorama data data 1222 1222 includes includes the complete the complete 360-degree 360-degree panoramas panoramas produced produced by the by the

panoramagenerator panorama generator1212 1212 forfor a agiven giveninput inputimage. image.TheThe storage storage manager manager 12101210 may may further further include include

training data training data 1224. The The training training data data 1224 1224may may full full panorama panorama images images obtained obtained from from an an image image

repository or repository or other other source source which whichcan canbebe used used by by training training manager manager 1208 1208 to train to train the panorama the panorama

generator. generator. In In some embodiments, some embodiments, thethe trainingdata training data1224 1224 cancan also also include include FOVFOV masks masks which which can can

be combinedwith be combined with thethe full full panoramas panoramas to create to create partial partial panoramas panoramas forduring for use use during training, training, as as

discussed above. discussed above.

[0060]

[0060] Each the components ofthe Each of 1202-1210 components1202-1210 of the of the panorama panorama generation systemsystem generation 1200 and 1200 and

their their corresponding elements(as corresponding elements (as shown shownininFIG. FIG. 12)12) maymay be communication be in in communication withanother with one one another

using any using any suitable suitable communication technologies.It will communication technologies. It willbeberecognized recognized thatalthough that although components components

1202-1210 and their 1202-1210 and their corresponding corresponding elements elements are are shown to be shown to be separate separate in in FIG. FIG. 12, 12, any any of of

components 1202-1210andand components 1202-1210 their their corresponding corresponding elements elements may may be combined be combined into fewer into fewer

components, suchasasinto components, such into aa single single facility facilityoror module, module,divided dividedinto more into morecomponents, or configured components, or configured

into into different differentcomponents as may components as mayserve serveaa particular particular embodiment. embodiment.

[0061]

[0061] The components1202-1210 The components 1202-1210 andand their their correspondingelements corresponding elements cancan comprise comprise

software, software, hardware, or both. hardware, or both. For Forexample, example, thethe components components 1202-1210 1202-1210 and corresponding and their their corresponding

elements cancomprise elements can compriseoneone or or more more instructions instructions stored stored on on a computer-readable a computer-readable storage storage medium medium

and executable by and executable byprocessors processorsofofone oneorormore more computing computing devices. devices. When When executed executed by the by one the or one or

more processors, more processors, the the computer-executable computer-executable instructionsof of instructions thepanorama the panorama generation generation system system 1200 1200

18

can causea aclient can cause clientdevice deviceand/or and/or a server a server device device to perform to perform the methods the methods described described herein. herein.

Alternatively, Alternatively,the thecomponents components 1202-1210 andtheir 1202-1210 and their corresponding corresponding elements elements can cancomprise comprise

hardware, such hardware, suchasas aaspecial special purpose purposeprocessing processingdevice devicetotoperform perform a certainfunction a certain functionororgroup group of of

functions. Additionally, the functions. Additionally, the components 1202-1210 and components 1202-1210 andtheir their corresponding corresponding elements elements can can

comprise comprise aa combination combinationofofcomputer-executable computer-executable instructions instructions andand hardware. hardware.

[0062]

[0062] Furthermore, the Furthermore, 1202-1210 components1202-1210 the components of the of the panorama panorama generation generation system system 1200 1200

may, for may, for example, example,bebe implemented implemented as orone as one or stand-alone more more stand-alone applications, applications, as more as one or one or more

modulesofofananapplication, modules application, as as one one or or more moreplug-ins, plug-ins, as as one one or or more morelibrary library functions functions or or functions functions

that may that be called may be called by by other other applications, applications, and/or and/orasasa acloud-computing cloud-computing model. Thus, the model. Thus, the

components 1202-1210 components 1202-1210 of the of the panorama panorama generation generation system system 1200 1200 may bemay be implemented implemented as a stand as a stand-

alone application, such alone application, as aa desktop such as or mobile desktop or mobile application. application. Furthermore, Furthermore,thethecomponents components 1202 1202-

1210 of the 1210 of the panorama panoramageneration generation system system 12001200 mayimplemented may be be implemented as more as one or one or more web-based web-based

applications hosted on applications hosted ona aremote remoteserver. server.Alternatively, Alternatively,or or additionally,thethecomponents additionally, components of the of the

panoramageneration panorama generationsystem system 1200 1200 maymay be implemented be implemented in a suite in a suite of mobile of mobile device device applications applications or or

"'apps." "apps."

[0063]

[0063] As shown, As shown,the thepanorama panorama generation generation system system 1200 1200 canimplemented can be be implemented as a as a single single

system. In other system. In other embodiments, embodiments,thethe panorama panorama generation generation systemsystem 1200 1200 can can be implemented be implemented in in

whole, or in whole, or in part, part, across across multiple multiple systems. systems. For For example, oneorormore example, one morefunctions functionsofofthe thepanorama panorama

generation system1200 generation system 1200cancanbebeperformed performed by one by one or more or more servers, servers, and and onemore one or or more functions functions of of

the panorama the generationsystem panorama generation system1200 1200 cancan be be performed performed by one by one or more or more client client devices. devices. TheThe one one or or

more servers and/or one or more client devices may generate, store, receive, and transmit any type more servers and/or one or more client devices may generate, store, receive, and transmit any type

of of data data used used by by the the panorama generationsystem panorama generation system1200, 1200,asasdescribed describedherein. herein.

19 moreclient one oror more the one devices can client devices include or can include implementatat or implement 2023200526 01 Feb 2023

[00641

[0064] In In one implementation, the one implementation,

least least aaportion portionofof thethe panorama panoramageneration generationsystem system 1200. In other 1200. In other implementations, the one implementations, the one or or more more

servers servers can can include include or implement at least implement at least aa portion portion of ofthe thepanorama generation system panorama generation system1200. 1200.ForFor

instance, instance, the the panorama generationsystem panorama generation system1200 1200 cancan include include an an application application running running on the on the oneone or or

more servers more servers or or aa portion portion of of the the panorama panoramageneration generationsystem system 1200 1200 can can be downloaded be downloaded from from the the

one or more one or moreservers. servers.Additionally Additionally or or alternatively,thethepanorama alternatively, panorama generation generation system system 1200 can 1200 can

include include aaweb web hosting hosting application application that that allows allows the client the client device(s) device(s) to interact to interact with hosted with content contentat hosted at

the one the oneorormore more server(s). server(s).

[0065]

[0065] For For example, clientdevice upona aclient example,upon deviceaccessing a webpage accessinga webpage or other or other web web application application

hostedatatthe hosted theone oneorormore more servers, servers, in one in one or more or more embodiments, embodiments, themore the one or oneservers or more canservers receive can receive

access to to one or more one or moredigital digital images images(e.g., (e.g., the the input input image data 1218, image data 1218, such suchasascamera cameraroll rollororanan

individual's personal individual's personal photos) photos) stored stored onclient on the the client device device or at another or at another storage storage location.location. Moreover, Moreover,

the client the client device devicecancan receive receive a request a request (i.e.,(i.e., via user via user input)input) to generate to generate a panorama a panorama for an inputfor an input

image andprovide image and providethe therequest requestto to the the one one or or more moreservers. servers. Upon Upon receiving receiving thethe request,the request, theone oneoror

more servers more servers can can automatically automaticallyperform performthe themethods methodsandand processes processes described described above above to generate to generate a a

full full 360-degree panorama.TheThe 360-degree panorama. one one or more or more servers servers can return can return the panorama the full full panorama to the to the client client

device fordisplay device for displayto to the the user. user.

[0066]

[0066] The server(s) and/or The server(s) device(s) may client device(s) and/or client communicate may communicate using using anyany communication communication

platforms and platforms and technologies technologies suitable suitable for for transporting transporting data dataand/or and/orcommunication signals, including communication signals, including

any knowncommunication any known communication technologies, technologies, devices, devices, media, media, and and protocols protocols supportive supportive of remote of remote data data

communications, examples communications, examples of which of which willwill be described be described in more in more detail detail below below with with respect respect to FIG. to FIG.

14. 14. In some someembodiments, embodiments,the the server(s) server(s) and/or and/or client client device(s) device(s) communicate communicate viaorone via one or more more

networks.A A networks. network network may include may include a single a single networknetwork or a collection or a collection of (such of networks networks as the(such Internet, as the Internet,

20

Feb 2023 aa corporate intranet,a avirtual corporate intranet, virtualprivate privatenetwork network (VPN), (VPN), a area a local localnetwork area network (LAN), a (LAN), wireless a wireless local local

network(WLAN), network (WLAN), a cellular a cellular network, network, a wide a wide areaarea network network (WAN), (WAN), a metropolitan a metropolitan area network area network

(MAN), (MAN), orora acombination combinationof of two two or or more more such such networks. networks. The The onemore one or or more networks networks 'M08bewill be `M08 will

2023200526 01

discussed in more discussed in detail below more detail with regard below with regard to to FIG. FIG. 14. 14.

[00671

[0067] server(s)maymay Theserver(s) The include include one one or more more hardware or hardware servers servers (e.g., hosts), with each hosts), (e.g.,each its with its

own computing own computing resources resources (e.g., (e.g., processors, processors, memory, memory, disk space, disk space, networking networking bandwidth, bandwidth, etc.) etc.)

which maybebesecurely which may securely divided divided between between multiple multiple customers customers (e.g.(e.g. client client devices), devices), each each of which of which

mayhost may hosttheir their own ownapplications applicationsononthe theserver(s). server(s). The Theclient clientdevice(s) device(s) may mayinclude includeoneone or or more more

personal computers, personal computers,laptop laptopcomputers, computers,mobile mobile devices, devices, mobile mobile phones, phones, tablets, tablets, special special purpose purpose

computers, TVs,ororother computers, TVs, othercomputing computing devices, devices, including including computing computing devices devices described described belowbelow with with

regardtotoFIG. regard FIG.14.14.

[0068]

[0068] FIGS. 1-12,thethe FIGS. 1-12, corresponding text,text, corresponding and the the examples, andexamples, a number aof provide provide number of different different

systems and devices systems and devices that that enables enables panorama panoramaextrapolation. extrapolation. InInaddition addition toto the theforegoing, foregoing,

embodiments can embodiments can alsobebedescribed also described in in terms terms of of flowchartscomprising flowcharts comprising acts acts andand steps steps in in a method a method

for for accomplishing a particular accomplishing a particular result. result. For For example, example, FIG. 13FIG. 13 illustrates illustrates a flowchart a flowchart of an exemplary of an exemplary

methodininaccordance method accordancewith withoneone or or more more embodiments. embodiments. The method The method described described in relation in relation to FIG.to FIG.

13 13 may be performed may be performedwith withfewer fewerorormore moresteps/acts steps/actsororthe the steps/acts steps/acts may be performed may be in differing performed in differing

orders. Additionally, orders. Additionally, thethe steps/acts steps/acts described described herein herein may bemay be repeated repeated or performed or performed in parallel in parallel with with

one anotheror or one another in in parallelwith parallel with different different instances instances ofsame of the the or same or similar similar steps/acts. steps/acts.

[0069]

[0069] FIG. 13 illustrates FIG. 13 illustrates aa flowchart flowchart 1300 of aa series 1300 of series of of acts acts in in aa method of extracting method of extracting

360-degree panoramas from 360-degree panoramas from narrow narrowfield field of of view viewimages imagesininaccordance accordancewith withone oneor ormore more

embodiments. embodiments. InIn oneor or one more more embodiments, embodiments, the method the method 1300 is1300 is performed performed in a digital in a digital medium medium

environment thatincludes environment that includes the the panorama panoramageneration generation system system 100. 100. The The method method 1300 1300 is is intended intended to to

21

be illustrative of one or more methods in accordance with the present disclosure and is not intended be illustrative of one or more methods in accordance with the present disclosure and is not intended

to limit to limit potential potential embodiments. embodiments.Alternative Alternative embodiments embodiments can include can include additional, additional, fewer, fewer, or or

different steps than different steps thanthose thosearticulated articulated in FIG. in FIG. 13. 13.

[00701

[0070] As illustrated As illustrated in in FIG. 13, the FIG. 13, the method 1300includes method 1300 includesan an actact 1302 1302 of obtaining of obtaining an an

input input image and guide. image and guide. For For example, example,the theuser user can can upload uploadananimage, image,download downloadan an image, image, provide provide a a

reference to the image (e.g., a URL, URI, etc.). As discussed, the input image is a narrow field of reference to the image (e.g., a URL, URI, etc.). As discussed, the input image is a narrow field of

view digital image. view digital image. For For example, example, aa narrow narrowFOV FOV image image may may include include an image an image with with a field a field of view of view

less less than than aafull full360360degrees. In In degrees. some embodiments, some embodiments, aauser userinterface interfacemay maybe beprovided provided through through which which

the user the user can can specify specify aa guide. guide.For For example, example, aa menu, list, ororother menu, list, othergraphic graphicelement elementmay may be be provided provided

which enablesthe which enables theuser usertotoselect select aa guide guide from froma alist list of of supported supported guides. guides. AsAsdiscussed, discussed,ininsome some

embodiments, embodiments, the the guide guide is a class is a class that that a a guide guide model model has beenhas beentotrained trained to identify. identify.

[00711

[0071] As illustrated As illustrated in in FIG. 13, the FIG. 13, the method 1300includes method 1300 includesan an actact 1304 1304 of generating of generating a a

panoramic projection panoramic projection of of the the input input image. image. In In some embodiments, generating some embodiments, generating the the panoramic panoramic

projection includes projection includes determining cameraparameters determining camera parametersassociated associatedwith withthe theinput input image imageand andgenerating generating

the panoramic the panoramicprojection projectionusing usingthethecamera camera parameters. parameters. In some In some embodiments, embodiments, the panoramic the panoramic

projection is projection is generated generated by by warping the input warping the input image basedononthe image based thecamera cameraparameters. parameters.

[0072]

[0072] As illustrated in FIG. 13, the method 1300 includes an act 1306 of generating, by a As illustrated in FIG. 13, the method 1300 includes an act 1306 of generating, by a

panorama generator,a a360-degree panorama generator, 360-degree panorama panorama basedbased onpanoramic on the the panoramic projection projection and theand the guide, guide,

wherein the wherein the panorama panoramagenerator generatoris isa aguided guidedco-modulation co-modulation generator generator network network trained trained to generate to generate

aa 360-degree panoramafrom 360-degree panorama from theinput the inputimage imagebased basedonon theguide. the guide.InInsome someembodiments, embodiments, generating generating

the 360-degree the panorama 360-degree panorama furtherincludes further includes determining determining a guide a guide vector vector based based on input on the the input imageimage

using using aa pre-trained pre-trained guide guidemodel model and and generating generating the 360-degree the 360-degree panorama panorama usingco-guided co using guided

modulationbased modulation basedon on thethe guide guide vector vector and panoramic and the the panoramic projection. projection. In someInembodiments, some embodiments,

22 generating generating the the 360-degree panoramafurther furtherincludes includesreceiving receivinga aguide guidetarget, target, determining determining aa guide guide 2023200526 01 Feb 2023

360-degree panorama

vector basedonon vector based thethe input input image image usingusing a pre-trained a pre-trained guide determining guide model, model, determining a target a target vector vector based based

on on aa difference difference between betweenthetheguide guide targetandand target thethe guide guide vector, vector, and and generating generating the 360-degree the 360-degree

panorama usingguided panorama using guided co-modulation co-modulation based based on the on the target target vector vector andand thethe panoramic panoramic projection. projection.

[0073]

[0073] In someembodiments, In some embodiments, the the panorama panorama generator generator is trained is trained by obtaining by obtaining training training

panorama images,creating panorama images, creatingpartial partial panorama images panorama images byby applying applying fieldofofview field viewmasks maskstotothe thetraining training

panorama images,generating, panorama images, generating,bybyananuntrained untrainedpanorama panorama generator generator network, network, generated generated panoramas panoramas

based on based onthe the partial partial panorama panoramaimages, images, evaluating evaluating the the generated generated panoramas panoramas by a discriminator by a discriminator

network using network usingthe thetraining trainingpanorama panorama images, images, and training and training the untrained the untrained panorama panorama generator generator

network and network andthe the discriminator discriminator network networkbased basedononthe theevaluation. evaluation. InIn some someembodiments, embodiments, evaluating evaluating

the generated the generated panoramas panoramas byby a adiscriminator discriminatornetwork network using using thethe trainingpanorama training panorama images, images, further further

includes applying a ahorizontal includes applying horizontalshift shifttotothethegenerated generated panoramas panoramas to create to create shifted shifted generated generated

panoramasand panoramas andevaluating evaluatingthetheshifted shiftedgenerated generatedpanoramas panoramas by abydiscriminator a discriminator network network usingusing the the

training panorama training images. panorama images.

[0074]

[0074] Embodiments Embodiments of of thethe present present disclosuremaymay disclosure comprise comprise or utilize or utilize a special a special purpose purpose

or general-purpose computer or general-purpose computer including including computer computer hardware, hardware, such such as, example, as, for for example, one orone moreor more

processors and processors and system systemmemory, memory, as discussed as discussed in greater in greater detail detail below. below. Embodiments Embodiments within within the the

scope ofthe scope of the present presentdisclosure disclosurealso alsoinclude includephysical physicalandand other other computer-readable computer-readable media media for for

carrying carrying ororstoring storingcomputer-executable computer-executable instructions instructions and/or and/or data data structures. structures. In particular, In particular, one or one or

more ofofthe more theprocesses processesdescribed described herein herein may may be implemented be implemented at leastatinleast part in as part as instructions instructions

embodied embodied inin a anon-transitory non-transitory computer-readable computer-readable medium mediumand andexecutable executablebyby oneone or more or more

computing devices(e.g., computing devices (e.g., any any of of the the media content access media content access devices devices described describedherein). herein). InIngeneral, general, aa

processor (e.g., processor (e.g., aamicroprocessor) microprocessor) receives receives instructions, instructions,from from aanon-transitory non-transitorycomputer-readable computer-readable

23 medium, (e.g., aa memory, memory,etc.), etc.), and and executes executesthose thoseinstructions, instructions, thereby performingone oneorormore more 2023200526 01 Feb 2023 medium, (e.g., thereby performing processes, including processes, one or including one or more ofthe more of the processes processes described described herein. herein.

[0075]

[0075] Computer-readable media Computer-readable media can can be any be any available available media media that be that can canaccessed be accessed by a by a

general general purpose or special purpose or special purpose purpose computer system. Computer-readable computer system. Computer-readable media mediathat thatstore store

computer-executable instructionsare computer-executable instructions arenon-transitory non-transitorycomputer-readable computer-readable storage storage media media (devices). (devices).

Computer-readable mediathat Computer-readable media thatcarry carrycomputer-executable computer-executable instructionsareare instructions transmission transmission

media. Thus,bybyway media. Thus, wayof of example, example, andand notnot limitation,embodiments limitation, embodiments of the of the disclosure disclosure cancan comprise comprise

at least two at least distinctly different two distinctly different kinds of computer-readable kinds of computer-readablemedia: media: non-transitory non-transitory computer computer-

readable storage readable storage media (devices) and media (devices) andtransmission transmissionmedia. media.

[0076]

[0076] Non-transitory computer-readable Non-transitory computer-readablestorage storagemedia media (devices) (devices) includes includes RAM, RAM, ROM, ROM,

EEPROM, CD-ROM, EEPROM, CD-ROM, solidsolid state state drives("SSDs") drives ("SSDs")(e.g., (e.g., based based on on RAM), Flash memory, RAM), Flash memory, phase- phase

change memory change memory ("PCM"), ("PCM"), other other types types of memory, of memory, otherother optical optical diskdisk storage,magnetic storage, magnetic disk disk storage storage

or or other other magnetic magnetic storage storage devices, devices, or orany anyother othermedium whichcan medium which can be be used usedto to store store desired desiredprogram program

code meansininthe code means theform formofofcomputer-executable computer-executable instructions instructions or or data data structuresand structures and which which cancan be be

accessed by aa general accessed by general purpose purpose or or special special purpose purpose computer. computer.

[00771

[0077] "network"is isdefined A "network" A as as defined oneone or more or more that that links links data data enable enable the transport the transport of of

electronic electronic data data between computersystems between computer systemsand/or and/ormodules modules and/or and/or other other electronicdevices. electronic devices.When When

information is transferred information is transferred or or provided providedover overa anetwork network or another or another communications communications connection connection

(either hardwired,wireless, (either hardwired, wireless, or or a combination a combination of hardwired of hardwired or wireless) or wireless) to a computer, to a computer, the computer the computer

properly views properly viewsthe theconnection connectionas as a transmission a transmission medium. medium. Transmissions Transmissions media media can can ainclude include a

networkand/or network and/ordata datalinks links which whichcan canbebeused usedtotocarry carrydesired desiredprogram program code code means means in the in the form form of of

computer-executable instructionsorordata computer-executable instructions datastructures structuresandand which which canaccessed can be be accessed by a general by a general

24

2023 purpose or purpose or special special purpose computer.Combinations purpose computer. Combinations of the of the above above should should also also be included be included within within

the scope the of computer-readable scope of media. computer-readable media. 2023200526 01 Feb

[0078]

[0078] Further, upon Further, reaching various upon reaching computersystem various computer components, systemcomponents, program meansmeans codecode program

in in the formofofcomputer-executable the form computer-executable instructions instructions or dataor data structures structures can be transferred can be transferred automatically automatically

from transmissionmedia from transmission mediato tonon-transitory non-transitorycomputer-readable computer-readable storage storage mediamedia (devices) (devices) (or vice (or vice

versa). versa). For example, computer-executable For example, computer-executableinstructions instructionsorordata datastructures structures received received over over aa network network

or or data data link link can can be be buffered buffered in inRAM withina anetwork RAM within networkinterface interfacemodule module (e.g.,a a"NIC"), (e.g., "NIC"),andand then then

eventually transferred to eventually transferred to computer systemRAM computer system RAM and/or and/or to less to less volatile volatile computer computer storage storage mediamedia

(devices) at aa computer (devices) at computersystem. system.Thus, Thus, it should it should be understood be understood that non-transitory that non-transitory computer computer-

readable storage readable storage media media(devices) (devices)can canbebeincluded includedin incomputer computer system system components components that (or that also also (or

even primarily)utilize even primarily) utilizetransmission transmission media. media.

[00791

[0079] Computer-executable Computer-executable instructions for for comprise, instructionscomprise, example, example, instructions instructions and data and data

which, when executed which, when executedatata aprocessor, processor, cause cause a ageneral-purpose general-purpose computer, computer, special special purpose purpose

computer, computer, ororspecial specialpurpose purpose processing processing device device to perform to perform a certain a certain functionfunction or groupor of group of

functions. In some functions. In someembodiments, embodiments, computer-executable computer-executable instructions instructions are executed are executed on a general on a general-

purpose computer purpose computer toto turn turn the the general-purpose general-purpose computer computer into into a aspecial special purpose purpose computer computer

implementing elements implementing elements of of the the disclosure. disclosure. The The computer computer executable executable instructions instructions may be, may for be, for

example, binaries, intermediate example, binaries, intermediate format formatinstructions instructions such suchasasassembly assembly language, language, or even or even source source

code. Althoughthethesubject code. Although subjectmatter matterhashasbeen been described described in in language language specific specific to to structuralfeatures structural features

and/or methodological and/or methodological acts,acts, it isitto is beto understood be understood that that the the subject subject matter indefined matter defined in the appended the appended

claims is not claims is not necessarily necessarily limited limited to to the the described features or described features or acts acts described above. Rather, described above. Rather,the the

described features and described features acts are and acts are disclosed disclosed as asexample example forms of implementing forms of implementingthe theclaims. claims.

25

Feb 2023

[00801

[0080] Those skilled inin the Those skilled will appreciate art will the art the disclosure that the appreciate that disclosure may practicedin in maybebepracticed

network computing network computing environments environments withwith manymany typestypes of computer of computer systemsystem configurations, configurations, including, including,

personal computers, personal computers, desktop computers, laptop desktop computers, laptop computers, message processors, computers, message processors, hand-held hand-held 2023200526 01

devices, devices, multi-processor systems, microprocessor-based multi-processor systems, microprocessor-basedor or programmable programmable consumer consumer electronics, electronics,

networkPCs, network PCs,minicomputers, minicomputers, mainframe mainframe computers, computers, mobilemobile telephones, telephones, PDAs, tablets, PDAs, tablets, pagers, pagers,

routers, switches, routers, and the switches, and the like. like. The The disclosure disclosure maymay also also be practiced be practiced in distributed in distributed systemsystem

environments where environments where localandand local remote remote computer computer systems, systems, whichwhich are linked are linked (either (either by hardwired by hardwired

data links, wireless data links, wirelessdata data links,or or links, by by a combination a combination of hardwired of hardwired and data and wireless wireless links)data links) through through

aa network, network, both bothperform performtasks. tasks.InIna distributed a distributedsystem systemenvironment, environment, program program modules modules may bemay be

located located in in both both local localand and remote remote memory storagedevices. memory storage devices.

[0081]

[0081] Embodiments ofofthe Embodiments thepresent canalso disclosure can presentdisclosure implemented alsobebeimplemented in cloud in cloud

computing environments.InInthis computing environments. thisdescription, description, "cloud computing"isis defined "cloud computing" defined as as aa model for enabling model for enabling

on-demand network on-demand network access access to shared to a a shared pool pool of configurable of configurable computing computing resources. resources. For example, For example,

cloud computingcancan cloud computing be be employed employed in marketplace in the the marketplace to ubiquitous to offer offer ubiquitous and convenient and convenient on- on

demand access demand access to to thethe shared shared poolpool of configurable of configurable computing computing resources. resources. Thepool The shared shared of pool of

configurable computing computingresources resourcescan canbeberapidly rapidlyprovisioned provisioned viavirtualization via virtualization and andreleased releasedwith with

low management low management effort effort oror serviceprovider service providerinteraction, interaction, and and then then scaled scaled accordingly. accordingly.

[0082]

[0082] A cloud-computing A cloud-computing model model can can be composed be composed of various of various characteristics characteristics such such as, as, for for

example, on-demand example, on-demand self-service, self-service, broad broad network network access,access, resource resource pooling,pooling, rapid elasticity, rapid elasticity,

measuredservice, measured service, and and so so forth. forth. A cloud-computingmodel A cloud-computing model cancan alsoexpose also expose various various servicemodels, service models,

such as, for such as, for example, example,Software Software asService as a a Service ("SaaS"), ("SaaS"), Platform Platform as a Service as a Service ("PaaS"), ("PaaS"), and and

Infrastructure as Infrastructure a Service as a Service ("IaaS"). ("IaaS"). A A cloud-computing cloud-computing model model canbe also can also be deployed deployed using using

different different deployment modelssuch deployment models suchasasprivate privatecloud, cloud, community community cloud, cloud, public public cloud, cloud, hybrid hybrid cloud, cloud,

26 and so forth. forth. InInthis andin inthetheclaims, descriptionand thisdescription claims, a "cloud-computing is an is an environment" 2023200526 01 Feb 2023 and so a "cloud-computing environment" environmentininwhich environment whichcloud computing cloudcomputing is employed. is employed.

[00831

[0083] FIG. FIG. 14 illustrates, ininblock 14 illustrates, diagram form, blockdiagram an exemplary form, an computing exemplarycomputing device 14001400 device

that may that beconfigured may be configuredto toperform perform one one or more or more of theofprocesses the processes described described above. above. One will One will

appreciate appreciate that that one one or ormore more computing devices such computing devices suchasas the the computing computingdevice device1400 1400maymay implement implement

the panorama the generationsystem. panorama generation system. As shown As shown by 14, by FIG. FIG.the14,computing the computing device device can can comprise comprise a a

processor 1402, processor 1402, memory memory 1404, 1404, oneone or or more more communication communication interfaces interfaces 1406, 1406, a storage a storage device device 1408, 1408,

and one or and one or more more I/O I/O devices/interfaces devices/interfaces 1410. 1410. InIn certain certain embodiments, embodiments,the thecomputing computing device device 1400 1400

can include fewer can include fewer or or more morecomponents components than than those those shown shown in FIG. in FIG. 14. Components 14. Components of computing of computing

device 1400shown device 1400 shownininFIG. FIG.14 14will willnow nowbe be described described in in additionaldetail. additional detail.

[0084]

[0084] In particular embodiments, In particular processor(s)14021402 embodiments, processor(s) includes includes hardware hardware for executing for executing

instructions, instructions, such such as as those those making upa acomputer making up computer program. program. Asexample, As an an example, andbynot and not wayby of way of

limitation, to execute limitation, to executeinstructions, instructions, processor(s) processor(s) 1402 1402 may retrieve may retrieve (orthe (or fetch) fetch) the instructions instructions from from

an internal register, an internal register,an aninternal internalcache, cache,memory 1404,orora astorage memory 1404, storagedevice device1408 1408 andand decode decode and and

execute them. InInvarious execute them. variousembodiments, embodiments, the the processor(s) processor(s) 14021402 may include may include one orone or central more more central

processing units processing units (CPUs), (CPUs),graphics graphics processing processing units units (GPUs), (GPUs), field field programmable programmable gate gate arrays arrays

(FPGAs), systemsononchip (FPGAs), systems chip(SoC), (SoC),or orother otherprocessor(s) processor(s)ororcombinations combinationsof of processors. processors.

[0085]

[0085] The computing device 1400 computing device 1400includes memory includes memory 1404, 1404, which which is coupled to to is coupled thethe

processor(s) 1402. processor(s) 1402. The Thememory memory 14041404 may may be be for used usedstoring for storing data, data, metadata, metadata, and programs and programs for for

execution by the execution by the processor(s). processor(s). The Thememory memory14041404 may include may include one orone or of more more of volatile volatile and non and non-

volatile volatilememories, memories,such as as such Random RandomAccess AccessMemory Memory ("RAM"), Read Only ("RAM"), Read Only Memory Memory ("ROM"), ("ROM"), a a

solid solid state statedisk disk("SSD"), ("SSD"), Flash, Flash, Phase Phase Change Memory Change Memory ("PCM"), ("PCM"), or other or other typestypes of data of data storage. storage.

The memory The memory 1404 1404 may may be internal be internal or distributed or distributed memory. memory.

27 one or include one further include or more communication morecommunication 2023200526 01 Feb 2023

[00861

[0086] The The computing device 1400 computing device canfurther 1400can

interfaces 1406. interfaces 1406. AAcommunication communication interface interface 1406 1406 can can include include hardware, hardware, software, software, or both. or both. The The

communication interface1406 communication interface 1406cancan provide provide or or one one more more interfacesfor interfaces forcommunication communication (such (such as, as, forfor

example, packet-basedcommunication) example, packet-based communication) between between the computing the computing device device and one and oneother or more or more other

computing devices1400 computing devices 1400 or or oneone or or more more networks. networks. Asexample As an an example and and not by not wayby of way of limitation, limitation,

communication interface1406 communication interface 1406 may may include include a network a network interface interface controller controller (NIC) (NIC) or or network network

adapter for adapter for communicating communicatingwithwith an Ethernet an Ethernet or other or other wire-based wire-based networknetwork or a wireless or a wireless NIC NIC

(WNIC) (WNIC) or or wirelessadapter wireless adapterforforcommunicating communicating with with a wireless a wireless network, network, such such as a WI-Fl. as a WI-FI. The The

computing device1400 computing device 1400 cancan further further include include a bus a bus 1412. 1412. The1412 The bus bus can 1412 can comprise comprise hardware, hardware,

software, software, or both both that thatcouples couples components ofcomputing components of computingdevice device 1400 1400 to to each each other. other.

[00871

[0087] The device computingdevice The computing 1400 1400 includes includes a storage a storage device 1408 1408 device includes includes storage storage for for

storing dataororinstructions. storing data instructions.As As an example, an example, and and not by not way by way of limitation, of limitation, storage storage device device 1408 can 1408 can

comprise comprise aa non-transitory non-transitory storage storage medium describedabove. medium described above.TheThe storage storage device device 1408 1408 may may include include

aa hard hard disk disk drive drive(HDD), flash memory, (HDD), flash memory, aa Universal Universal Serial Serial Bus Bus (USB) (USB)drive driveororaa combination combinationthese these

or or other other storage storage devices. devices. The computingdevice The computing device 1400 1400 alsoalso includes includes one one or more or more inputinput or output or output

("I/O") devices/interfaces 1410, ("I/O") devices/interfaces whichare 1410, which are provided providedtotoallow allowa auser usertotoprovide provideinput inputtoto(such (suchasas

user strokes), user strokes), receive receive output output from, from, and and otherwise transfer data otherwise transfer data to to and and from from the the computing device computing device

1400. These TheseI/OI/O devices/interfaces1410 devices/interfaces 1410 maymay include include a mouse, a mouse, keypad keypad or a keyboard, or a keyboard, a toucha touch

screen, camera, optical screen, camera, opticalscanner, scanner,network network interface, interface, modem, modem, other other known known I/O 1/O ordevices devices a or a

combination ofsuch combination of suchI/O 1/O devices/interfaces devices/interfaces 1410. 1410. The Thetouch touchscreen screenmay maybe be activatedwith activated with a a stylus stylus

or a finger. or a finger.

[0088]

[0088] The I/Odevices/interfaces The I/O devices/interfaces 1410 1410may may include include oneone or more or more devices devices for presenting for presenting

output toaauser, output to user,including, including,butbut notnot limited limited to, to, a graphics a graphics engine, engine, a display a display (e.g., (e.g., a display a display screen),screen),

28

Feb 2023 one or more one or moreoutput outputdrivers drivers(e.g., (e.g., display display drivers), drivers), one one or or more audio speakers, more audio speakers, and andone oneorormore more

audio drivers. audio drivers. InIncertain certainembodiments, embodiments,I/O I/O devices/interfaces devices/interfaces 1410 1410 is configured is configured to provide to provide

graphical datatotoa adisplay graphical data display forfor presentation presentation to a to a user. user. The graphical The graphical data maydata may be representative be representative of of 2023200526 01

one ormore one or more graphical graphical useruser interfaces interfaces and/or and/or anygraphical any other other graphical content content as as amay may serve serve a particular particular

implementation. implementation.

[0089]

[0089] In the In specification,embodiments foregoing specification, the foregoing embodiments have beendescribed havebeen describedwith referencetoto withreference

specific specific exemplary embodiments exemplary embodiments thereof. thereof. Various Various embodiments embodiments are described are described with reference with reference to to

details details discussed discussed herein, herein,and andthe theaccompanying drawingsillustrate accompanying drawings illustrate the the various various embodiments. The embodiments. The

description above and description above anddrawings drawingsareare illustrative ofofone illustrative oneorormore more embodiments embodiments andnot and are aretonot be to be

construed as limiting. construed as limiting. Numerous Numerousspecific specificdetails details are aredescribed described totoprovide providea thorough a thorough

understanding ofvarious understanding of various embodiments. embodiments.

[0090]

[0090] Embodiments maymay Embodiments include include other other specific specific forms forms without without departing its its fromfrom departing spiritoror spirit

essential essential characteristics. characteristics.The The described described embodiments aretotobebeconsidered embodiments are consideredininall all respects respects only only as as

illustrative illustrativeand andnot notrestrictive. ForFor restrictive. example, thethe example, methods methodsdescribed describedherein hereinmay may be beperformed with performed with

less less or or more steps/acts or more steps/acts or the the steps/acts steps/actsmay may be be performed in differing performed in differing orders. orders. Additionally, Additionally, the the

steps/acts describedherein steps/acts described herein maymay be repeated be repeated or performed or performed in parallel in parallel with one with oneoranother another or in parallel in parallel

withdifferent with differentinstances instancesof of thethe same same or similar or similar steps/acts. steps/acts. Theofscope The scope of the invention the invention is, therefore, is, therefore,

indicated by by the the appended claimsrather appended claims rather than than by by the the foregoing foregoing description. description. All All changes changes that that come come

within the within the meaning andrange meaning and rangeofofequivalency equivalencyofofthe theclaims claimsare aretoto be be embraced embracedwithin withintheir theirscope. scope.

[0091]

[0091] In the In the various various embodiments described embodiments described above, above, unless unless specificallynoted specifically noted otherwise, otherwise,

disjunctive language disjunctive language suchsuch as phrase as the the phrase "at one "at least leastof one A, B,oforA,C," B,isorintended C," is intended to be understood to be understood

to mean to meaneither eitherA, A, B, B, or or or C, C, any or any combination combination thereof thereof (e.g., A,(e.g., A, B,C).and/or B, and/or C).disjunctive As such, As such, disjunctive

29

language language isisnot notintended intended to,to, nornor should should it beit understood be understood to, imply to, imply that a that givena embodiment given embodiment requires requires

at least one at least ofA,A,atatleast one of oneofofB,B,or or least one at at leastoneone least of of C each C to to each be present. be present.

30

Claims (18)

CLAIMS 08 Jul 2025 We Claim:

1. A method comprising: obtaining an input image and guide; generating a panoramic projection of the input image; and generating, by a panorama generator, a 360-degree panorama based on the panoramic 2023200526

projection and the guide, wherein the panorama generator is a guided co-modulation generator network trained to generate a 360-degree panorama using guided co-modulation based on a target vector and the panoramic projection, wherein the target vector is determined based on a difference between a guide target and a guide vector, and wherein the guide vector is determined based on the input image using a pre-trained guide model.

2. The method of claim 1, wherein generating a panoramic projection of the input image, further comprises: determining camera parameters associated with the input image; and generating the panoramic projection using the camera parameters.

3. The method of claim 1, wherein generating a panoramic projection of the input image, further comprises: obtaining camera parameters associated with the input image; and warping the input image based on the camera parameters.

4. The method of claim 1, wherein generating, by a panorama generator, a 360-degree panorama based on the panoramic projection and the guide, wherein the panorama generator is a guided co-modulation generator network trained to generate a 360-degree panorama from the input image based on the guide, further comprises: determining a guide vector based on the input image using a pre-trained guide model; and generating the 360-degree panorama using guided co-modulation based on the guide vector and the panoramic projection.

5. The method of claim 1, wherein the guide is a class that a guide model has been trained 08 Jul 2025

to identify.

6. The method of claim 1, wherein the input image is a narrow field of view digital image.

7. The method of claim 1, wherein the panorama generator is trained by: 2023200526

obtaining training panorama images; creating partial panorama images by applying field of view masks to the training panorama images; generating, by an untrained panorama generator network, generated panoramas based on the partial panorama images; evaluating the generated panoramas by a discriminator network using the training panorama images; and training the untrained panorama generator network and the discriminator network based on the evaluation.

8. The method of claim 7, wherein evaluating the generated panoramas by a discriminator network using the training panorama images, further comprises: applying a horizontal shift to the generated panoramas to create shifted generated panoramas; and evaluating the shifted generated panoramas by a discriminator network using the training panorama images.

9. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising: obtaining an input image and guide; generating a panoramic projection of the input image; and generating, by a panorama generator, a 360-degree panorama based on the panoramic projection and the guide, wherein the panorama generator is a guided co-modulation generator network trained to generate a 360-degree panorama using guided co-modulation based on a target vector and the panoramic projection, wherein the target vector is determined based on a difference between a guide target and a guide vector, and wherein the guide vector is determined 08 Jul 2025 based on the input image using a pre-trained guide model.

10. The non-transitory computer-readable medium of claim 9, wherein the operation of generating a panoramic projection of the input image, further comprises: determining camera parameters associated with the input image; and 2023200526

generating the panoramic projection using the camera parameters.

11. The non-transitory computer-readable medium of claim 9, wherein the operation of generating a panoramic projection of the input image, further comprises: obtaining camera parameters associated with the input image; and warping the input image based on the camera parameters.

12. The non-transitory computer-readable medium of claim 9, wherein the operation of generating, by a panorama generator, a 360-degree panorama based on the panoramic projection and the guide, wherein the panorama generator is a guided co-modulation generator network trained to generate a 360-degree panorama from the input image based on the guide, further comprises: determining a guide vector based on the input image using a pre-trained guide model; and generating the 360-degree panorama using guided co-modulation based on the guide vector and the panoramic projection.

13. The non-transitory computer-readable medium of claim 9, wherein the guide is a class that a guide model has been trained to identify.

14. The non-transitory computer-readable medium of claim 9, wherein the input image is a narrow field of view digital image.

15. The non-transitory computer-readable medium of claim 9, wherein the panorama generator is trained by: obtaining training panorama images; creating partial panorama images by applying field of view masks to the training 08 Jul 2025 panorama images; generating, by an untrained panorama generator network, generated panoramas based on the partial panorama images; evaluating the generated panoramas by a discriminator network using the training panorama images; and 2023200526 training the untrained panorama generator network and the discriminator network based on the evaluation.

16. The non-transitory computer-readable medium of claim 15, wherein evaluating the generated panoramas by a discriminator network using the training panorama images, further comprises: applying a horizontal shift to the generated panoramas to create shifted generated panoramas; and evaluating the shifted generated panoramas by a discriminator network using the training panorama images.

17. A system comprising: a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising: obtaining an input image and guide; generating a panoramic projection of the input image; and generating, by a panorama generator, a 360-degree panorama based on the panoramic projection and the guide, wherein the panorama generator is a guided co- modulation generator network trained to generate a 360-degree panorama using guided co-modulation based on a target vector and the panoramic projection, wherein the target vector is determined based on a difference between a guide target and a guide vector and wherein the guide vector is determined based on the input image using a pre-trained guide model.

18. The system of claim 17, wherein the operation of generating a panoramic projection of 08 Jul 2025

the input image, further comprises: determining camera parameters associated with the input image; and generating the panoramic projection using the camera parameters. 2023200526